Nov 02 01:22:00.879627 ip-10-0-136-53 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 02 01:22:00.879639 ip-10-0-136-53 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 02 01:22:00.879648 ip-10-0-136-53 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 02 01:22:00.879975 ip-10-0-136-53 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 02 01:22:10.905507 ip-10-0-136-53 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 02 01:22:10.905523 ip-10-0-136-53 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 79678c34a51e4ad894e1a089f15b6fcf --
Nov 02 01:24:23.442663 ip-10-0-136-53 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 01:24:23.756945 ip-10-0-136-53 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:24:23.756945 ip-10-0-136-53 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 01:24:23.756945 ip-10-0-136-53 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:24:23.756945 ip-10-0-136-53 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:24:23.760809 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760727    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:24:23.760809 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760811    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:24:23.760809 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760814    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:24:23.760912 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760817    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:24:23.760912 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760820    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:24:23.760912 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760823    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:24:23.760912 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760826    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:24:23.760912 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760829    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:24:23.760912 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760831    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:24:23.760912 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760834    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:24:23.760912 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760836    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:24:23.760912 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760839    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:24:23.760912 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760841    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:24:23.760912 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760844    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:24:23.760912 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760846    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:24:23.760912 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760849    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:24:23.760912 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760851    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:24:23.760912 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760855    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:24:23.760912 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760858    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:24:23.760912 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760867    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:24:23.760912 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760870    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:24:23.760912 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760873    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:24:23.760912 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760875    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:24:23.761820 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760878    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:24:23.761820 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760881    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:24:23.761820 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760883    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:24:23.761820 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760886    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:24:23.761820 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760889    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:24:23.761820 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760891    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:24:23.761820 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760894    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:24:23.761820 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760897    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:24:23.761820 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760899    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:24:23.761820 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760902    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:24:23.761820 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760904    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:24:23.761820 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760907    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:24:23.761820 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760910    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:24:23.761820 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760913    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:24:23.761820 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760916    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:24:23.761820 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760919    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:24:23.761820 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760924    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:24:23.761820 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760928    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:24:23.761820 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760930    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:24:23.762259 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760934    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:24:23.762259 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760938    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:24:23.762259 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760942    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:24:23.762259 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760945    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:24:23.762259 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760948    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:24:23.762259 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760951    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:24:23.762259 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760954    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:24:23.762259 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760957    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:24:23.762259 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760960    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:24:23.762259 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760962    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:24:23.762259 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760965    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:24:23.762259 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760967    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:24:23.762259 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760970    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:24:23.762259 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760972    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:24:23.762259 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760975    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:24:23.762259 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760978    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:24:23.762259 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760980    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:24:23.762259 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760983    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:24:23.762259 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760985    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:24:23.762713 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760988    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:24:23.762713 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760990    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:24:23.762713 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760993    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:24:23.762713 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760995    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:24:23.762713 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.760997    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:24:23.762713 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761000    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:24:23.762713 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761003    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:24:23.762713 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761006    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:24:23.762713 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761009    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:24:23.762713 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761011    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:24:23.762713 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761014    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:24:23.762713 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761017    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:24:23.762713 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761019    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:24:23.762713 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761021    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:24:23.762713 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761024    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:24:23.762713 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761026    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:24:23.762713 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761029    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:24:23.762713 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761032    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:24:23.762713 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761034    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:24:23.762713 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761036    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:24:23.763175 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761039    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:24:23.763175 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761042    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:24:23.763175 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761046    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:24:23.763175 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761050    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:24:23.763175 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761054    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:24:23.763175 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761058    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:24:23.763175 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761061    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:24:23.763175 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761064    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:24:23.763175 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761067    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:24:23.763175 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761069    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:24:23.763175 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761072    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:24:23.763175 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761074    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:24:23.763175 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761077    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:24:23.763175 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761079    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:24:23.763175 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761082    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:24:23.763175 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761084    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:24:23.763175 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761087    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:24:23.763175 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761090    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:24:23.763175 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761093    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:24:23.763666 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761096    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:24:23.763666 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761099    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:24:23.763666 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761102    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:24:23.763666 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761104    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:24:23.763666 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761108    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:24:23.763666 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761111    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:24:23.763666 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761903    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:24:23.763666 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761910    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:24:23.763666 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761913    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:24:23.763666 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761916    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:24:23.763666 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761921    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:24:23.763666 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761924    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:24:23.763666 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761927    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:24:23.763666 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761930    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:24:23.763666 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761933    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:24:23.763666 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761936    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:24:23.763666 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761945    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:24:23.763666 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761948    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:24:23.763666 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761952    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:24:23.764125 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761954    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:24:23.764125 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761957    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:24:23.764125 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761960    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:24:23.764125 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761962    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:24:23.764125 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761965    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:24:23.764125 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761967    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:24:23.764125 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761970    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:24:23.764125 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761972    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:24:23.764125 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761975    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:24:23.764125 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761977    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:24:23.764125 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761980    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:24:23.764125 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761983    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:24:23.764125 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761985    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:24:23.764125 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761989    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:24:23.764125 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761991    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:24:23.764125 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761994    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:24:23.764125 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761997    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:24:23.764125 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.761999    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:24:23.764125 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762002    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:24:23.764598 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762004    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:24:23.764598 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762006    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:24:23.764598 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762009    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:24:23.764598 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762012    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:24:23.764598 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762015    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:24:23.764598 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762017    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:24:23.764598 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762020    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:24:23.764598 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762022    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:24:23.764598 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762025    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:24:23.764598 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762027    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:24:23.764598 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762030    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:24:23.764598 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762033    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:24:23.764598 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762036    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:24:23.764598 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762039    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:24:23.764598 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762041    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:24:23.764598 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762044    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:24:23.764598 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762046    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:24:23.764598 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762048    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:24:23.764598 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762051    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:24:23.764598 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762054    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:24:23.765077 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762056    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:24:23.765077 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762059    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:24:23.765077 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762062    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:24:23.765077 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762064    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:24:23.765077 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762066    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:24:23.765077 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762069    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:24:23.765077 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762072    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:24:23.765077 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762074    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:24:23.765077 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762077    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:24:23.765077 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762079    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:24:23.765077 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762082    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:24:23.765077 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762084    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:24:23.765077 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762087    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:24:23.765077 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762089    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:24:23.765077 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762091    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:24:23.765077 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762094    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:24:23.765077 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762097    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:24:23.765077 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762100    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:24:23.765077 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762102    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:24:23.765077 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762104    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:24:23.765561 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762107    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:24:23.765561 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762110    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:24:23.765561 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762118    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:24:23.765561 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762121    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:24:23.765561 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762124    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:24:23.765561 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762127    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:24:23.765561 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762129    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:24:23.765561 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762132    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:24:23.765561 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762134    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:24:23.765561 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762136    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:24:23.765561 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762140    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:24:23.765561 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762142    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:24:23.765561 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762144    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:24:23.765561 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762147    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:24:23.765561 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762149    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:24:23.765561 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762152    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:24:23.765561 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762155    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:24:23.765561 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762159    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:24:23.765561 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762163    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:24:23.766008 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762166    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:24:23.766008 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762170    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:24:23.766008 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762172    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:24:23.766008 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762175    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:24:23.766008 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762179    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:24:23.766008 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762181    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:24:23.766008 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762184    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:24:23.766008 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762186    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:24:23.766008 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762189    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:24:23.766008 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762191    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:24:23.766008 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762194    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:24:23.766008 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762197    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:24:23.766008 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762199    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:24:23.766008 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762202    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:24:23.766008 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762204    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:24:23.766008 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762728    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:24:23.766008 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762734    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:24:23.766008 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762737    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:24:23.766008 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762740    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:24:23.766008 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762743    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:24:23.766499 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762746    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:24:23.766499 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762748    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:24:23.766499 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762751    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:24:23.766499 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762753    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:24:23.766499 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762756    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:24:23.766499 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762759    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:24:23.766499 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762762    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:24:23.766499 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762764    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:24:23.766499 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762767    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:24:23.766499 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762769    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:24:23.766499 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762772    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:24:23.766499 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762775    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:24:23.766499 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762778    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:24:23.766499 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762780    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:24:23.766499 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762783    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:24:23.766499 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762786    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:24:23.766499 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762788    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:24:23.766499 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762791    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:24:23.766499 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762794    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:24:23.766499 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762796    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:24:23.766980 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762799    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:24:23.766980 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762802    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:24:23.766980 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762804    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:24:23.766980 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762807    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:24:23.766980 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762810    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:24:23.766980 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762812    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:24:23.766980 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762815    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:24:23.766980 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762817    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:24:23.766980 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762820    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:24:23.766980 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762822    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:24:23.766980 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762825    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:24:23.766980 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762828    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:24:23.766980 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762830    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:24:23.766980 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762833    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:24:23.766980 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762836    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:24:23.766980 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762838    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:24:23.766980 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762841    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:24:23.766980 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762843    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:24:23.766980 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762845    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:24:23.767431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762848    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:24:23.767431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762850    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:24:23.767431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762853    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:24:23.767431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762858    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:24:23.767431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762861    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:24:23.767431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762863    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:24:23.767431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762866    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:24:23.767431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762870    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:24:23.767431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762873    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:24:23.767431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762875    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:24:23.767431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762878    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:24:23.767431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762881    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:24:23.767431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762883    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:24:23.767431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762886    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:24:23.767431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762888    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:24:23.767431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762891    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:24:23.767431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762893    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:24:23.767431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762895    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:24:23.767431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762898    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:24:23.767885 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762900    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:24:23.767885 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762903    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:24:23.767885 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762905    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:24:23.767885 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762908    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:24:23.767885 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762911    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:24:23.767885 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762914    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:24:23.767885 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762916    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:24:23.767885 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762919    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:24:23.767885 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762921    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:24:23.767885 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762924    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:24:23.767885 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762926    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:24:23.767885 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762928    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:24:23.767885 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762931    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:24:23.767885 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762933    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:24:23.767885 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762936    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:24:23.767885 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762939    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:24:23.767885 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762943    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:24:23.767885 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762945    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:24:23.767885 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762948    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:24:23.767885 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762950    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:24:23.768411 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762953    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:24:23.768411 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762956    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:24:23.768411 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762958    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:24:23.768411 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762961    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:24:23.768411 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762964    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:24:23.768411 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762968    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:24:23.768411 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762972    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:24:23.768411 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762974    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:24:23.768411 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762977    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:24:23.768411 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762979    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:24:23.768411 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762982    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:24:23.768411 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762984    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:24:23.768411 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762987    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:24:23.768411 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762989    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:24:23.768411 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762992    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:24:23.768411 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762994    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:24:23.768411 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.762999    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:24:23.768411 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.763002    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:24:23.768411 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.763005    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:24:23.768858 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.763008    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:24:23.768858 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.763010    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:24:23.768858 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.763013    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:24:23.768858 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.763015    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:24:23.768858 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765468    2574 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 01:24:23.768858 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765483    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 01:24:23.768858 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765490    2574 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 01:24:23.768858 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765495    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 01:24:23.768858 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765500    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 01:24:23.768858 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765503    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 01:24:23.768858 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765508    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 01:24:23.768858 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765513    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 01:24:23.768858 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765516    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 01:24:23.768858 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765519    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 01:24:23.768858 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765523    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 01:24:23.768858 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765527    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 01:24:23.768858 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765530    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 01:24:23.768858 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765533    2574 flags.go:64] FLAG: --cgroup-root=""
Nov 02 01:24:23.768858 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765535    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 01:24:23.768858 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765538    2574 flags.go:64] FLAG: --client-ca-file=""
Nov 02 01:24:23.768858 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765541    2574 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 01:24:23.768858 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765544    2574 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 01:24:23.768858 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765550    2574 flags.go:64] FLAG: --cluster-domain=""
Nov 02 01:24:23.769402 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765552    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 01:24:23.769402 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765556    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 01:24:23.769402 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765559    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 01:24:23.769402 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765563    2574 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 01:24:23.769402 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765567    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 01:24:23.769402 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765570    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 01:24:23.769402 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765573    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 01:24:23.769402 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765576    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 01:24:23.769402 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765579    2574 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 01:24:23.769402 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765583    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 01:24:23.769402 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765586    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 01:24:23.769402 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765589    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 01:24:23.769402 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765592    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 01:24:23.769402 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765596    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 01:24:23.769402 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765599    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 01:24:23.769402 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765602    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 01:24:23.769402 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765605    2574 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 01:24:23.769402 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765608    2574 flags.go:64] FLAG: --enable-server="true"
Nov 02 01:24:23.769402 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765613    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 01:24:23.769402 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765617    2574 flags.go:64] FLAG: --event-burst="100"
Nov 02 01:24:23.769402 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765620    2574 flags.go:64] FLAG: --event-qps="50"
Nov 02 01:24:23.769402 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765624    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 01:24:23.769402 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765627    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 01:24:23.769402 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765630    2574 flags.go:64] FLAG: --eviction-hard=""
Nov 02 01:24:23.770040 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765635    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 01:24:23.770040 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765638    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 01:24:23.770040 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765641    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 01:24:23.770040 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765644    2574 flags.go:64] FLAG: --eviction-soft=""
Nov 02 01:24:23.770040 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765647    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 01:24:23.770040 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765651    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 01:24:23.770040 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765653    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 01:24:23.770040 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765656    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 01:24:23.770040 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765659    2574 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 01:24:23.770040 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765662    2574 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 01:24:23.770040 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765665    2574 flags.go:64] FLAG: --feature-gates=""
Nov 02 01:24:23.770040 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765668    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 01:24:23.770040 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765671    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 01:24:23.770040 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765674    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 01:24:23.770040 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765677    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 01:24:23.770040 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765681    2574 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 01:24:23.770040 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765684    2574 flags.go:64] FLAG: --help="false"
Nov 02 01:24:23.770040 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765687    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:24:23.770040 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765712    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 01:24:23.770040 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765717    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 01:24:23.770040 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765720    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 01:24:23.770040 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765724    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 01:24:23.770040 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765727    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 01:24:23.770620 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765730    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 01:24:23.770620 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765733    2574 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 01:24:23.770620 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765736    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 01:24:23.770620 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765739    2574 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 01:24:23.770620 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765742    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 01:24:23.770620 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765745    2574 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 01:24:23.770620 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765748    2574 flags.go:64] FLAG: --kube-reserved=""
Nov 02 01:24:23.770620 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765751    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 01:24:23.770620 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765754    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 01:24:23.770620 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765757    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 01:24:23.770620 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765760    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 01:24:23.770620 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765762    2574 flags.go:64] FLAG: --lock-file=""
Nov 02 01:24:23.770620 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765765    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 01:24:23.770620 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765768    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 01:24:23.770620 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765771    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 01:24:23.770620 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765777    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 01:24:23.770620 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765780    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 01:24:23.770620 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765783    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 01:24:23.770620 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765786    2574 flags.go:64] FLAG: --logging-format="text"
Nov 02 01:24:23.770620 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765789    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 01:24:23.770620 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765792    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 01:24:23.770620 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765795    2574 flags.go:64] FLAG: --manifest-url=""
Nov 02 01:24:23.770620 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765798    2574 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 01:24:23.770620 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765803    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 01:24:23.770620 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765806    2574 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 01:24:23.771287 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765810    2574 flags.go:64] FLAG: --max-pods="110"
Nov 02 01:24:23.771287 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765813    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 01:24:23.771287 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765816    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 01:24:23.771287 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765820    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 01:24:23.771287 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765822    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 01:24:23.771287 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765825    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 01:24:23.771287 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765828    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 01:24:23.771287 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765831    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 01:24:23.771287 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765839    2574 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 01:24:23.771287 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765843    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 01:24:23.771287 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765846    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 01:24:23.771287 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765849    2574 flags.go:64] FLAG: --pod-cidr=""
Nov 02 01:24:23.771287 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765851    2574 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 01:24:23.771287 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765854    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 01:24:23.771287 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765857    2574 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 01:24:23.771287 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765861    2574 flags.go:64] FLAG: --port="10250"
Nov 02 01:24:23.771287 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765864    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 01:24:23.771287 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765866    2574 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0aae5c789d6075945"
Nov 02 01:24:23.771287 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765870    2574 flags.go:64] FLAG: --qos-reserved=""
Nov 02 01:24:23.771287 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765872    2574 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 01:24:23.771287 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765875    2574 flags.go:64] FLAG: --register-node="true"
Nov 02 01:24:23.771287 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765878    2574 flags.go:64] FLAG: --register-with-taints=""
Nov 02 01:24:23.771287 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765882    2574 flags.go:64] FLAG: --registry-burst="10"
Nov 02 01:24:23.771287 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765884    2574 flags.go:64] FLAG: --registry-qps="5"
Nov 02 01:24:23.771287 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765887    2574 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 01:24:23.771287 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765890    2574 flags.go:64] FLAG: --reserved-memory=""
Nov 02 01:24:23.772167 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765894    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 01:24:23.772167 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765896    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 01:24:23.772167 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765899    2574 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 01:24:23.772167 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765902    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 01:24:23.772167 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765905    2574 flags.go:64] FLAG: --runonce="false"
Nov 02 01:24:23.772167 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765908    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 01:24:23.772167 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765913    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 01:24:23.772167 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765916    2574 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 01:24:23.772167 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765919    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 01:24:23.772167 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765922    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 01:24:23.772167 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765926    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 01:24:23.772167 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765929    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 01:24:23.772167 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765932    2574 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 01:24:23.772167 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765936    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 01:24:23.772167 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765939    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 01:24:23.772167 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765942    2574 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 01:24:23.772167 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765946    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 01:24:23.772167 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765949    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 01:24:23.772167 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765952    2574 flags.go:64] FLAG: --system-cgroups=""
Nov 02 01:24:23.772167 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765955    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 01:24:23.772167 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765961    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 01:24:23.772167 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765964    2574 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 01:24:23.772167 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765966    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 01:24:23.772167 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765971    2574 flags.go:64] FLAG: --tls-min-version=""
Nov 02 01:24:23.772167 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765974    2574 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 01:24:23.773212 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765977    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 01:24:23.773212 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765980    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 01:24:23.773212 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765983    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 01:24:23.773212 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765986    2574 flags.go:64] FLAG: --v="2"
Nov 02 01:24:23.773212 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765990    2574 flags.go:64] FLAG: --version="false"
Nov 02 01:24:23.773212 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765994    2574 flags.go:64] FLAG: --vmodule=""
Nov 02 01:24:23.773212 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.765998    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 01:24:23.773212 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.766001    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 01:24:23.773212 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766147    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:24:23.773212 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766151    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:24:23.773212 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766154    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:24:23.773212 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766157    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:24:23.773212 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766160    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:24:23.773212 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766164    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:24:23.773212 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766167    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:24:23.773212 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766170    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:24:23.773212 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766172    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:24:23.773212 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766175    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:24:23.773212 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766178    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:24:23.773212 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766181    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:24:23.773212 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766184    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:24:23.773212 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766186    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:24:23.774083 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766189    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:24:23.774083 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766193    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:24:23.774083 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766195    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:24:23.774083 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766198    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:24:23.774083 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766201    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:24:23.774083 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766203    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:24:23.774083 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766205    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:24:23.774083 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766208    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:24:23.774083 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766210    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:24:23.774083 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766213    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:24:23.774083 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766216    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:24:23.774083 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766218    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:24:23.774083 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766221    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:24:23.774083 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766224    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:24:23.774083 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766227    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:24:23.774083 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766230    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:24:23.774083 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766232    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:24:23.774083 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766235    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:24:23.774083 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766237    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:24:23.774083 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766240    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:24:23.774645 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766242    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:24:23.774645 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766245    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:24:23.774645 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766247    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:24:23.774645 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766251    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:24:23.774645 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766254    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:24:23.774645 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766257    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:24:23.774645 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766259    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:24:23.774645 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766262    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:24:23.774645 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766265    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:24:23.774645 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766268    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:24:23.774645 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766270    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:24:23.774645 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766273    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:24:23.774645 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766290    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:24:23.774645 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766294    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:24:23.774645 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766296    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:24:23.774645 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766299    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:24:23.774645 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766303    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:24:23.774645 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766307    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:24:23.774645 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766309    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:24:23.775103 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766312    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:24:23.775103 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766314    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:24:23.775103 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766317    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:24:23.775103 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766319    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:24:23.775103 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766322    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:24:23.775103 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766324    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:24:23.775103 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766327    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:24:23.775103 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766330    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:24:23.775103 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766332    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:24:23.775103 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766335    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:24:23.775103 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766337    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:24:23.775103 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766339    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:24:23.775103 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766342    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:24:23.775103 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766344    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:24:23.775103 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766347    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:24:23.775103 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766349    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:24:23.775103 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766353    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:24:23.775103 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766355    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:24:23.775103 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766358    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:24:23.775586 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766360    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:24:23.775586 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766363    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:24:23.775586 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766365    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:24:23.775586 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766368    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:24:23.775586 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766371    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:24:23.775586 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766375    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:24:23.775586 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766378    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:24:23.775586 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766382    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:24:23.775586 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766385    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:24:23.775586 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766388    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:24:23.775586 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766390    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:24:23.775586 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766392    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:24:23.775586 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766395    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:24:23.775586 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766397    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:24:23.775586 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766400    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:24:23.775586 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766403    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:24:23.775586 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766405    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:24:23.775586 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766408    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:24:23.775586 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766410    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:24:23.776049 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766413    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:24:23.776049 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766415    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:24:23.776049 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766418    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:24:23.776049 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766420    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:24:23.776049 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766423    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:24:23.776049 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766425    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:24:23.776049 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766428    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:24:23.776049 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766431    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:24:23.776049 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766433    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:24:23.776049 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766435    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:24:23.776049 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766439    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:24:23.776049 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766442    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:24:23.776049 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766445    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:24:23.776049 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766447    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:24:23.776049 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766450    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:24:23.776049 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.766478    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:24:23.776445 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766513    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:24:23.776445 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766519    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:24:23.776445 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766523    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:24:23.776445 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766526    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:24:23.776445 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766530    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:24:23.776445 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766532    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:24:23.776445 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766535    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:24:23.776445 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766537    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:24:23.776445 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766540    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:24:23.776445 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766542    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:24:23.776445 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766545    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:24:23.776445 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766547    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:24:23.776445 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766550    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:24:23.776445 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766553    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:24:23.776445 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766555    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:24:23.776445 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766558    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:24:23.776445 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766560    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:24:23.776445 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766562    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:24:23.776445 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766565    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:24:23.776880 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766569    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:24:23.776880 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766572    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:24:23.776880 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766575    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:24:23.776880 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766578    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:24:23.776880 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766581    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:24:23.776880 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766584    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:24:23.776880 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766586    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:24:23.776880 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766589    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:24:23.776880 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766593    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:24:23.776880 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766596    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:24:23.776880 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766598    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:24:23.776880 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766601    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:24:23.776880 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766603    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:24:23.776880 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766606    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:24:23.776880 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766608    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:24:23.776880 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766611    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:24:23.776880 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766614    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:24:23.776880 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766616    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:24:23.776880 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766619    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:24:23.777413 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766621    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:24:23.777413 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766624    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:24:23.777413 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766626    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:24:23.777413 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766629    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:24:23.777413 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766631    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:24:23.777413 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766634    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:24:23.777413 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766636    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:24:23.777413 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766639    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:24:23.777413 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766642    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:24:23.777413 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766645    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:24:23.777413 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766647    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:24:23.777413 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766650    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:24:23.777413 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766652    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:24:23.777413 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766655    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:24:23.777413 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766657    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:24:23.777413 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766659    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:24:23.777413 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766662    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:24:23.777413 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766664    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:24:23.777413 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766667    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:24:23.777854 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766669    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:24:23.777854 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766671    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:24:23.777854 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766674    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:24:23.777854 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766676    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:24:23.777854 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766679    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:24:23.777854 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766681    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:24:23.777854 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766684    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:24:23.777854 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766686    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:24:23.777854 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766688    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:24:23.777854 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766691    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:24:23.777854 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766694    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:24:23.777854 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766697    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:24:23.777854 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766700    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:24:23.777854 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766702    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:24:23.777854 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766705    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:24:23.777854 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766707    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:24:23.777854 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766710    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:24:23.777854 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766712    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:24:23.777854 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766715    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:24:23.777854 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766717    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:24:23.778372 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766720    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:24:23.778372 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766722    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:24:23.778372 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766725    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:24:23.778372 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766727    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:24:23.778372 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766729    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:24:23.778372 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766732    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:24:23.778372 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766735    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:24:23.778372 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766737    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:24:23.778372 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766739    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:24:23.778372 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766742    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:24:23.778372 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766744    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:24:23.778372 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766747    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:24:23.778372 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766749    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:24:23.778372 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766752    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:24:23.778372 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766754    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:24:23.778372 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766757    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:24:23.778372 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766759    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:24:23.778372 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766762    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:24:23.778372 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766764    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:24:23.778372 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766767    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:24:23.778835 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766769    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:24:23.778835 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766771    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:24:23.778835 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766774    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:24:23.778835 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766777    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:24:23.778835 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766780    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:24:23.778835 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766782    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:24:23.778835 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766784    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:24:23.778835 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766788    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:24:23.778835 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.766790    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:24:23.778835 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.775159    2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 01:24:23.778835 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.775177    2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 01:24:23.778835 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.775734    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 01:24:23.778835 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.775741    2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 01:24:23.778835 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775809    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:24:23.778835 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775816    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:24:23.778835 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775820    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:24:23.778835 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775823    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:24:23.778835 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775826    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:24:23.778835 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775828    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:24:23.778835 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775831    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:24:23.779323 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775834    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:24:23.779323 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775836    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:24:23.779323 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775839    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:24:23.779323 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775841    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:24:23.779323 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775844    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:24:23.779323 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775846    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:24:23.779323 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775849    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:24:23.779323 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775852    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:24:23.779323 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775854    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:24:23.779323 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775856    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:24:23.779323 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775859    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:24:23.779323 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775862    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:24:23.779323 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775864    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:24:23.779323 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775867    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:24:23.779323 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775870    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:24:23.779323 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775873    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:24:23.779323 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775877    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:24:23.779323 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775879    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:24:23.779323 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775882    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:24:23.779773 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775885    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:24:23.779773 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775887    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:24:23.779773 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775889    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:24:23.779773 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775892    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:24:23.779773 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775895    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:24:23.779773 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775897    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:24:23.779773 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775900    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:24:23.779773 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775902    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:24:23.779773 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775905    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:24:23.779773 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775907    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:24:23.779773 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775910    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:24:23.779773 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775912    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:24:23.779773 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775915    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:24:23.779773 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775917    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:24:23.779773 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775920    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:24:23.779773 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775922    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:24:23.779773 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775925    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:24:23.779773 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775927    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:24:23.779773 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775930    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:24:23.779773 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775933    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:24:23.780239 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775937    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:24:23.780239 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775939    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:24:23.780239 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775942    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:24:23.780239 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775945    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:24:23.780239 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775948    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:24:23.780239 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775953    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:24:23.780239 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775957    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:24:23.780239 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775960    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:24:23.780239 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775963    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:24:23.780239 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775966    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:24:23.780239 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775969    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:24:23.780239 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775971    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:24:23.780239 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775974    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:24:23.780239 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775977    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:24:23.780239 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775979    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:24:23.780239 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775982    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:24:23.780239 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775985    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:24:23.780239 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775988    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:24:23.780239 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775990    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:24:23.780786 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775993    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:24:23.780786 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775995    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:24:23.780786 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.775998    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:24:23.780786 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776000    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:24:23.780786 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776012    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:24:23.780786 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776015    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:24:23.780786 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776018    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:24:23.780786 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776020    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:24:23.780786 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776023    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:24:23.780786 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776026    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:24:23.780786 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776028    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:24:23.780786 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776031    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:24:23.780786 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776034    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:24:23.780786 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776036    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:24:23.780786 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776040    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:24:23.780786 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776043    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:24:23.780786 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776045    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:24:23.780786 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776048    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:24:23.780786 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776052    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:24:23.781264 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776056    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:24:23.781264 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776059    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:24:23.781264 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776062    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:24:23.781264 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776065    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:24:23.781264 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776067    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:24:23.781264 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776070    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:24:23.781264 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776073    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:24:23.781264 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776075    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:24:23.781264 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776078    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:24:23.781264 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776080    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:24:23.781264 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776083    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:24:23.781264 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776086    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:24:23.781264 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776088    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:24:23.781264 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776091    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:24:23.781264 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776093    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:24:23.781264 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776096    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:24:23.781264 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776098    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:24:23.781264 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776101    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:24:23.781264 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776103    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:24:23.781264 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776106    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:24:23.781745 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776108    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:24:23.781745 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776111    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:24:23.781745 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.776144    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:24:23.781745 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776313    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:24:23.781745 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776320    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:24:23.781745 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776323    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:24:23.781745 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776326    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:24:23.781745 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776329    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:24:23.781745 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776333    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:24:23.781745 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776336    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:24:23.781745 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776340    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:24:23.781745 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776342    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:24:23.781745 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776346    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:24:23.781745 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776350    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:24:23.781745 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776352    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:24:23.781745 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776355    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:24:23.782124 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776358    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:24:23.782124 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776361    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:24:23.782124 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776364    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:24:23.782124 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776366    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:24:23.782124 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776369    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:24:23.782124 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776371    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:24:23.782124 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776374    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:24:23.782124 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776377    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:24:23.782124 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776379    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:24:23.782124 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776382    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:24:23.782124 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776384    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:24:23.782124 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776387    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:24:23.782124 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776390    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:24:23.782124 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776392    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:24:23.782124 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776394    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:24:23.782124 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776397    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:24:23.782124 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776400    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:24:23.782124 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776402    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:24:23.782124 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776405    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:24:23.782124 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776407    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:24:23.782597 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776410    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:24:23.782597 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776412    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:24:23.782597 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776415    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:24:23.782597 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776417    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:24:23.782597 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776420    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:24:23.782597 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776424    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:24:23.782597 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776426    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:24:23.782597 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776429    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:24:23.782597 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776432    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:24:23.782597 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776434    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:24:23.782597 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776437    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:24:23.782597 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776439    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:24:23.782597 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776442    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:24:23.782597 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776445    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:24:23.782597 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776448    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:24:23.782597 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776451    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:24:23.782597 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776453    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:24:23.782597 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776456    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:24:23.782597 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776459    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:24:23.783054 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776462    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:24:23.783054 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776465    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:24:23.783054 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776467    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:24:23.783054 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776469    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:24:23.783054 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776472    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:24:23.783054 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776474    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:24:23.783054 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776477    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:24:23.783054 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776480    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:24:23.783054 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776483    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:24:23.783054 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776485    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:24:23.783054 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776488    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:24:23.783054 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776490    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:24:23.783054 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776492    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:24:23.783054 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776495    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:24:23.783054 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776498    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:24:23.783054 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776500    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:24:23.783054 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776504    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:24:23.783054 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776508    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:24:23.783054 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776511    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:24:23.783521 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776515    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:24:23.783521 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776518    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:24:23.783521 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776521    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:24:23.783521 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776523    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:24:23.783521 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776526    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:24:23.783521 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776529    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:24:23.783521 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776532    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:24:23.783521 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776534    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:24:23.783521 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776536    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:24:23.783521 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776539    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:24:23.783521 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776541    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:24:23.783521 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776544    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:24:23.783521 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776547    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:24:23.783521 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776549    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:24:23.783521 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776553    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:24:23.783521 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776557    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:24:23.783521 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776560    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:24:23.783521 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776563    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:24:23.783521 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776566    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:24:23.783521 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776568    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:24:23.783996 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776571    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:24:23.783996 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776574    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:24:23.783996 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776576    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:24:23.783996 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776579    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:24:23.783996 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776581    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:24:23.783996 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776584    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:24:23.783996 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776586    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:24:23.783996 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776589    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:24:23.783996 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776591    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:24:23.783996 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776594    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:24:23.783996 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776596    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:24:23.783996 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776598    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:24:23.783996 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776601    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:24:23.783996 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776604    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:24:23.783996 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776607    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:24:23.783996 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.776630    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:24:23.784391 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776663    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:24:23.784391 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776667    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:24:23.784391 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776671    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:24:23.784391 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776674    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:24:23.784391 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776677    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:24:23.784391 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776680    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:24:23.784391 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776683    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:24:23.784391 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776686    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:24:23.784391 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776689    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:24:23.784391 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776692    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:24:23.784391 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776694    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:24:23.784391 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776697    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:24:23.784391 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776700    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:24:23.784391 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776704    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:24:23.784391 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776708    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:24:23.784391 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776711    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:24:23.784391 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776713    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:24:23.784391 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776716    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:24:23.784391 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776719    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:24:23.784858 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776721    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:24:23.784858 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776724    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:24:23.784858 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776726    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:24:23.784858 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776729    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:24:23.784858 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776731    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:24:23.784858 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776734    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:24:23.784858 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776736    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:24:23.784858 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776738    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:24:23.784858 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776741    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:24:23.784858 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776743    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:24:23.784858 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776748    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:24:23.784858 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776752    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:24:23.784858 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776756    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:24:23.784858 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776758    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:24:23.784858 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776761    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:24:23.784858 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776764    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:24:23.784858 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776766    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:24:23.784858 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776769    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:24:23.784858 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776771    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:24:23.784858 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776774    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:24:23.785431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776776    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:24:23.785431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776779    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:24:23.785431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776781    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:24:23.785431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776784    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:24:23.785431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776786    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:24:23.785431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776789    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:24:23.785431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776791    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:24:23.785431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776794    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:24:23.785431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776797    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:24:23.785431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776799    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:24:23.785431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776802    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:24:23.785431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776805    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:24:23.785431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776807    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:24:23.785431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776809    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:24:23.785431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776812    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:24:23.785431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776814    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:24:23.785431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776817    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:24:23.785431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776819    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:24:23.785431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776822    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:24:23.785431 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776824    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:24:23.786081 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776827    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:24:23.786081 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776829    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:24:23.786081 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776834    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:24:23.786081 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776837    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:24:23.786081 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776840    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:24:23.786081 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776843    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:24:23.786081 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776846    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:24:23.786081 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776849    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:24:23.786081 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776851    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:24:23.786081 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776854    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:24:23.786081 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776857    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:24:23.786081 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776859    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:24:23.786081 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776861    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:24:23.786081 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776864    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:24:23.786081 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776866    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:24:23.786081 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776869    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:24:23.786081 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776871    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:24:23.786081 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776874    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:24:23.786081 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776876    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:24:23.786081 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776878    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:24:23.786720 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776881    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:24:23.786720 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776884    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:24:23.786720 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776887    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:24:23.786720 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776889    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:24:23.786720 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776892    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:24:23.786720 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776895    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:24:23.786720 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776897    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:24:23.786720 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776899    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:24:23.786720 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776902    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:24:23.786720 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776905    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:24:23.786720 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776908    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:24:23.786720 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776910    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:24:23.786720 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776913    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:24:23.786720 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776915    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:24:23.786720 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776918    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:24:23.786720 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776921    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:24:23.786720 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776924    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:24:23.786720 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776927    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:24:23.786720 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776929    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:24:23.786720 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776933    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:24:23.787216 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776936    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:24:23.787216 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776938    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:24:23.787216 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776941    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:24:23.787216 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776943    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:24:23.787216 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776945    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:24:23.787216 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776948    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:24:23.787216 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:23.776950    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:24:23.787216 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.777010    2574 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 01:24:23.787216 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.780209    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 01:24:23.787216 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.780985    2574 server.go:1014] "Starting client certificate rotation"
Nov 02 01:24:23.787216 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.781079    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 01:24:23.787216 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.781640    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 01:24:23.787216 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.782770    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 01:24:23.787216 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.785294    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 01:24:23.795144 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.795123    2574 log.go:25] "Validated CRI v1 runtime API"
Nov 02 01:24:23.797983 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.797964    2574 log.go:25] "Validated CRI v1 image API"
Nov 02 01:24:23.798883 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.798858    2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 01:24:23.801362 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.801341    2574 fs.go:135] Filesystem UUIDs: map[1cc9a6b5-91ef-4870-8a42-6bec1d4eedd9:/dev/nvme0n1p3 30057d81-0a35-49a8-a663-c830ef847dab:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Nov 02 01:24:23.801416 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.801363    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 01:24:23.806381 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.806240    2574 manager.go:217] Machine: {Timestamp:2026-11-02 01:24:23.80574379 +0000 UTC m=+0.267261455 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098489 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2068d50a12027b12b614dba11ef171 SystemUUID:ec2068d5-0a12-027b-12b6-14dba11ef171 BootID:79678c34-a51e-4ad8-94e1-a089f15b6fcf Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:64:df:cd:fb:93 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:64:df:cd:fb:93 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:cd:dd:5c:1b:bf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 01:24:23.806381 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.806377    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 01:24:23.806526 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.806460    2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 01:24:23.810244 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.810047    2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 01:24:23.810409 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.810247    2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-53.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 01:24:23.810888 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.810877    2574 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 01:24:23.810918 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.810891    2574 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 01:24:23.810918 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.810904    2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 01:24:23.811541 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.811531    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 01:24:23.812200 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.812189    2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 01:24:23.815649 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.815637    2574 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 01:24:23.816307 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.816297    2574 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 01:24:23.816968 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.816959    2574 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 01:24:23.816997 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.816973    2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 01:24:23.816997 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.816984    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 01:24:23.816997 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.816994    2574 kubelet.go:405] "Adding apiserver pod source"
Nov 02 01:24:23.817068 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.817002    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 01:24:23.817425 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.817411    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 01:24:23.819413 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.819387    2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 01:24:23.823488 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.823473    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 01:24:23.824161 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.824149    2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 01:24:23.824195 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.824180    2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 01:24:23.824383 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.824373    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 01:24:23.824427 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.824393    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 01:24:23.824427 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.824404    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 01:24:23.824427 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.824416    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 01:24:23.824512 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.824427    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 01:24:23.824512 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.824437    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 01:24:23.824850 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.824842    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 01:24:23.824882 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.824852    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 01:24:23.824882 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.824859    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 01:24:23.824882 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.824864    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 01:24:23.824882 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.824874    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 01:24:23.824882 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.824884    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 01:24:23.826133 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.826123    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 01:24:23.826133 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.826134    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 01:24:23.830082 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.830060    2574 server.go:1263] "Started kubelet"
Nov 02 01:24:23.830177 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.830085    2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 01:24:23.830177 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.830120    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 01:24:23.830251 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.830199    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 01:24:23.830876 ip-10-0-136-53 systemd[1]: Started Kubernetes Kubelet.
Nov 02 01:24:23.831177 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.831157    2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 01:24:23.832853 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.832840    2574 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 01:24:23.835142 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:23.835124    2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 01:24:23.835553 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.835535    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 01:24:23.836027 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.836010    2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 01:24:23.836192 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:23.836164    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-53.us-west-1.compute.internal\" not found"
Nov 02 01:24:23.836525 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.836499    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 01:24:23.836611 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.836564    2574 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 01:24:23.836611 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.836582    2574 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 01:24:23.836748 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.836722    2574 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 01:24:23.836748 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.836738    2574 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 01:24:23.837198 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.837170    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 01:24:23.837198 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.837197    2574 factory.go:55] Registering systemd factory
Nov 02 01:24:23.837315 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.837204    2574 factory.go:223] Registration of the systemd container factory successfully
Nov 02 01:24:23.837458 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.837444    2574 factory.go:153] Registering CRI-O factory
Nov 02 01:24:23.837500 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.837464    2574 factory.go:223] Registration of the crio container factory successfully
Nov 02 01:24:23.837529 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.837507    2574 factory.go:103] Registering Raw factory
Nov 02 01:24:23.837555 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.837531    2574 manager.go:1199] Started watching for new ooms in manager
Nov 02 01:24:23.838497 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.838480    2574 manager.go:319] Starting recovery of all containers
Nov 02 01:24:23.850994 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.850973    2574 manager.go:324] Recovery completed
Nov 02 01:24:23.852217 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:23.852192    2574 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Nov 02 01:24:23.855101 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.855089    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:24:23.856683 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.856664    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:24:23.856767 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.856704    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:24:23.856767 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.856729    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:24:23.857312 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.857300    2574 cpu_manager.go:226] "Starting" policy="none"
Nov 02 01:24:23.857312 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.857312    2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 01:24:23.857395 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.857328    2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 01:24:23.857395 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.857303    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:24:23.858044 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.858030    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:24:23.858124 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.858060    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:24:23.858124 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.858075    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:24:23.859453 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.859440    2574 policy_none.go:50] "Start"
Nov 02 01:24:23.859502 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.859457    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 01:24:23.859502 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.859471    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 01:24:23.861435 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.861424    2574 policy_none.go:44] "Start"
Nov 02 01:24:23.902950 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.902907    2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 01:24:23.909450 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.909436    2574 manager.go:341] "Starting Device Plugin manager"
Nov 02 01:24:23.909548 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:23.909535    2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 01:24:23.909548 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.909548    2574 server.go:87] "Starting device plugin registration server"
Nov 02 01:24:23.909800 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.909789    2574 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 01:24:23.909858 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.909804    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 01:24:23.909967 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.909952    2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 01:24:23.910053 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.910042    2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 01:24:23.910053 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.910052    2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 01:24:23.910558 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:23.910536    2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 01:24:23.910632 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:23.910593    2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-53.us-west-1.compute.internal\" not found"
Nov 02 01:24:23.966054 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.965878    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 01:24:23.968242 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.967421    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 01:24:23.968242 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.967449    2574 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 01:24:23.968242 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.967483    2574 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 01:24:23.968242 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:23.967519    2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 02 01:24:23.981356 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:23.981329    2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-53.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 01:24:23.981356 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:23.981336    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-53.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 01:24:23.982111 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:23.981329    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-53.us-west-1.compute.internal.18e415ee10dfe313  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-53.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-53.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 01:24:23.829807891 +0000 UTC m=+0.291325557,LastTimestamp:2026-11-02 01:24:23.829807891 +0000 UTC m=+0.291325557,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-53.us-west-1.compute.internal,}"
Nov 02 01:24:24.009945 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.009879    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:24:24.010837 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.010819    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:24:24.010939 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.010858    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:24:24.010939 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.010875    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:24:24.010939 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.010908    2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:24:24.062726 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.062688    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 01:24:24.062905 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.062737    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l4pnp"
Nov 02 01:24:24.069073 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.068950    2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal"]
Nov 02 01:24:24.069998 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.069979    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:24:24.072962 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.072946    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:24:24.073016 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.072979    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:24:24.073016 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.072990    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:24:24.075656 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.075642    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:24:24.075806 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.075788    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:24:24.075851 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.075821    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:24:24.076415 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.076398    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:24:24.076491 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.076442    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:24:24.076491 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.076453    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:24:24.076491 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.076403    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:24:24.076491 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.076480    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:24:24.076606 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.076494    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:24:24.078442 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.078428    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:24:24.078497 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.078455    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:24:24.079172 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.079156    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:24:24.079299 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.079184    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:24:24.079299 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.079197    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:24:24.098972 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:24.098946    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-53.us-west-1.compute.internal\" not found" node="ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:24:24.105044 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:24.105021    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-53.us-west-1.compute.internal\" not found" node="ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:24:24.138857 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.138827    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14b7d72c2c054e228c464e14811dfb58-config\") pod \"kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal\" (UID: \"14b7d72c2c054e228c464e14811dfb58\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:24:24.138857 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.138858    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/48935a76d1ff2f699968aa919b673e4d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal\" (UID: \"48935a76d1ff2f699968aa919b673e4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:24:24.138857 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.138884    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48935a76d1ff2f699968aa919b673e4d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal\" (UID: \"48935a76d1ff2f699968aa919b673e4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:24:24.138857 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.138885    2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:24:24.237225 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.237190    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:24:24.238764 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.238745    2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:24:24.239014 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.238997    2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:24:24.240534 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.238977    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48935a76d1ff2f699968aa919b673e4d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal\" (UID: \"48935a76d1ff2f699968aa919b673e4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:24:24.240656 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.240584    2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:24:24.240656 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:24.240616    2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-53.us-west-1.compute.internal\": node \"ip-10-0-136-53.us-west-1.compute.internal\" not found"
Nov 02 01:24:24.240772 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.239056    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48935a76d1ff2f699968aa919b673e4d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal\" (UID: \"48935a76d1ff2f699968aa919b673e4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:24:24.240772 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.240731    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14b7d72c2c054e228c464e14811dfb58-config\") pod \"kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal\" (UID: \"14b7d72c2c054e228c464e14811dfb58\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:24:24.240872 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.240786    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/48935a76d1ff2f699968aa919b673e4d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal\" (UID: \"48935a76d1ff2f699968aa919b673e4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:24:24.240872 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.240836    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/48935a76d1ff2f699968aa919b673e4d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal\" (UID: \"48935a76d1ff2f699968aa919b673e4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:24:24.240972 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.240872    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14b7d72c2c054e228c464e14811dfb58-config\") pod \"kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal\" (UID: \"14b7d72c2c054e228c464e14811dfb58\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:24:24.337231 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.337146    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:24:24.396216 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.396193    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:24:24.397008 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.396995    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:24:24.401310 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.401292    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:24:24.407878 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.407857    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:24:24.456368 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.456326    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:24:24.780907 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.780829    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 01:24:24.781412 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.780977    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 01:24:24.781412 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.780985    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 01:24:24.781412 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:24.780978    2574 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a379cc44211674ec397976e30fa324b1-fb94896701b40ffa.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-136-53.us-west-1.compute.internal.18e415ee127ac879\": read tcp 10.0.136.53:49568->3.18.235.62:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-136-53.us-west-1.compute.internal.18e415ee127ac879  default   5413 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-53.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-53.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-53.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 01:24:23 +0000 UTC,LastTimestamp:2026-11-02 01:24:24.01088141 +0000 UTC m=+0.472399077,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-53.us-west-1.compute.internal,}"
Nov 02 01:24:24.817844 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.817815    2574 apiserver.go:52] "Watching apiserver"
Nov 02 01:24:24.835768 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:24.835740    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 01:24:25.067780 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.067684    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 01:19:24 +0000 UTC" deadline="2028-06-21 06:19:33.169031274 +0000 UTC"
Nov 02 01:24:25.067780 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.067722    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14332h55m8.101312969s"
Nov 02 01:24:25.090632 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.090607    2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 01:24:25.090962 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.090941    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-stb4n","openshift-dns/node-resolver-bflth","openshift-image-registry/node-ca-9lvcx","openshift-multus/multus-additional-cni-plugins-kmzfl","kube-system/konnectivity-agent-vlg9k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal","openshift-multus/multus-n5rzm","openshift-multus/network-metrics-daemon-rfwwn","openshift-network-operator/iptables-alerter-mjlgj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978","openshift-network-diagnostics/network-check-target-w4jf8","openshift-ovn-kubernetes/ovnkube-node-4vjq9","kube-system/global-pull-secret-syncer-9kqsh"]
Nov 02 01:24:25.093388 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.093368    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mjlgj"
Nov 02 01:24:25.095517 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.095495    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.097579 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.097560    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bflth"
Nov 02 01:24:25.099665 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.099648    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9lvcx"
Nov 02 01:24:25.101880 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.101864    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:24:25.103984 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.103964    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vlg9k"
Nov 02 01:24:25.106122 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.106104    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:25.106212 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:25.106180    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfwwn" podUID="9eb605c8-f995-49ae-a1b8-06aa55160b13"
Nov 02 01:24:25.108356 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.108337    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.110492 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.110477    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.112700 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.112686    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:24:25.112787 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:25.112737    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4jf8" podUID="4240f2d9-dd99-4ca9-8447-7706cebcfa08"
Nov 02 01:24:25.115043 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.115026    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:24:25.117260 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.117242    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:25.117373 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:25.117343    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kqsh" podUID="26cf53e3-9cf3-4ad5-afe2-5c241f6749ad"
Nov 02 01:24:25.138517 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.138492    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 01:24:25.144984 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.144960    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fswd\" (UniqueName: \"kubernetes.io/projected/b0d87141-a1da-4d64-b01e-e66d8ba5e9ac-kube-api-access-9fswd\") pod \"node-resolver-bflth\" (UID: \"b0d87141-a1da-4d64-b01e-e66d8ba5e9ac\") " pod="openshift-dns/node-resolver-bflth"
Nov 02 01:24:25.145107 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.144995    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-sysconfig\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.145107 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145017    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-sys\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.145107 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145038    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdxpz\" (UniqueName: \"kubernetes.io/projected/9eb605c8-f995-49ae-a1b8-06aa55160b13-kube-api-access-zdxpz\") pod \"network-metrics-daemon-rfwwn\" (UID: \"9eb605c8-f995-49ae-a1b8-06aa55160b13\") " pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:25.145107 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145059    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-var-lib-cni-bin\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.145107 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145081    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbldg\" (UniqueName: \"kubernetes.io/projected/65dfab40-fb6d-46d7-98f1-8243e194c176-kube-api-access-dbldg\") pod \"iptables-alerter-mjlgj\" (UID: \"65dfab40-fb6d-46d7-98f1-8243e194c176\") " pod="openshift-network-operator/iptables-alerter-mjlgj"
Nov 02 01:24:25.145296 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145128    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-var-lib-openvswitch\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.145296 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145169    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/929a4d2e-02fa-4d5f-ace0-0ad3d41614b7-konnectivity-ca\") pod \"konnectivity-agent-vlg9k\" (UID: \"929a4d2e-02fa-4d5f-ace0-0ad3d41614b7\") " pod="kube-system/konnectivity-agent-vlg9k"
Nov 02 01:24:25.145296 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145235    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-lib-modules\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.145296 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145258    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-tuned\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.145586 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145311    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:24:25.145586 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145382    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-etc-selinux\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:24:25.145586 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145410    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-run-openvswitch\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.145586 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145431    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/929a4d2e-02fa-4d5f-ace0-0ad3d41614b7-agent-certs\") pod \"konnectivity-agent-vlg9k\" (UID: \"929a4d2e-02fa-4d5f-ace0-0ad3d41614b7\") " pod="kube-system/konnectivity-agent-vlg9k"
Nov 02 01:24:25.145586 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145457    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-sysctl-d\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.145586 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145480    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-registration-dir\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:24:25.145586 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145508    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d352c5c1-44a0-4929-a43a-30c723958ce7-host\") pod \"node-ca-9lvcx\" (UID: \"d352c5c1-44a0-4929-a43a-30c723958ce7\") " pod="openshift-image-registry/node-ca-9lvcx"
Nov 02 01:24:25.145586 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145532    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-run-ovn\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.145586 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145573    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-run-ovn-kubernetes\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.145891 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145630    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6238ba5-f475-4e61-94d8-7d7bc13b2458-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:24:25.145891 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145663    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-system-cni-dir\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.145891 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145686    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-multus-socket-dir-parent\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.145891 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145713    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-hostroot\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.145891 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145732    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-multus-daemon-config\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.145891 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145747    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-slash\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.145891 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145759    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ef8451f5-e54e-42e3-aad7-16e13fb70d81-ovnkube-script-lib\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.145891 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145773    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d352c5c1-44a0-4929-a43a-30c723958ce7-serviceca\") pod \"node-ca-9lvcx\" (UID: \"d352c5c1-44a0-4929-a43a-30c723958ce7\") " pod="openshift-image-registry/node-ca-9lvcx"
Nov 02 01:24:25.145891 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145794    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzgv6\" (UniqueName: \"kubernetes.io/projected/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-kube-api-access-xzgv6\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.145891 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145849    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-kubernetes\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.145891 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145880    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-dbus\") pod \"global-pull-secret-syncer-9kqsh\" (UID: \"26cf53e3-9cf3-4ad5-afe2-5c241f6749ad\") " pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:25.146441 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145909    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65dfab40-fb6d-46d7-98f1-8243e194c176-iptables-alerter-script\") pod \"iptables-alerter-mjlgj\" (UID: \"65dfab40-fb6d-46d7-98f1-8243e194c176\") " pod="openshift-network-operator/iptables-alerter-mjlgj"
Nov 02 01:24:25.146441 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145944    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-log-socket\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.146441 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145970    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6238ba5-f475-4e61-94d8-7d7bc13b2458-cnibin\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:24:25.146441 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.145986    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-multus-cni-dir\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.146441 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146004    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-socket-dir\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:24:25.146441 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146030    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.146441 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146051    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-run-multus-certs\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.146441 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146071    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-modprobe-d\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.146441 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146103    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-sysctl-conf\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.146441 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146121    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-var-lib-kubelet\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.146441 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146134    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-var-lib-cni-multus\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.146441 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146151    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj9pc\" (UniqueName: \"kubernetes.io/projected/ef8451f5-e54e-42e3-aad7-16e13fb70d81-kube-api-access-gj9pc\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.146441 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146174    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ef8451f5-e54e-42e3-aad7-16e13fb70d81-env-overrides\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.146441 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146196    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6238ba5-f475-4e61-94d8-7d7bc13b2458-system-cni-dir\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:24:25.146441 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146215    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6238ba5-f475-4e61-94d8-7d7bc13b2458-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:24:25.146441 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146235    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-run-k8s-cni-cncf-io\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.147317 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146372    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-kubelet\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.147317 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146407    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs\") pod \"network-metrics-daemon-rfwwn\" (UID: \"9eb605c8-f995-49ae-a1b8-06aa55160b13\") " pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:25.147317 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146430    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p47d\" (UniqueName: \"kubernetes.io/projected/7407afb5-5371-46c7-93ad-896d94a6f015-kube-api-access-9p47d\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:24:25.147317 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146457    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-systemd-units\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.147317 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146484    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-run-systemd\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.147317 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146518    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6238ba5-f475-4e61-94d8-7d7bc13b2458-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:24:25.147317 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146543    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d87141-a1da-4d64-b01e-e66d8ba5e9ac-tmp-dir\") pod \"node-resolver-bflth\" (UID: \"b0d87141-a1da-4d64-b01e-e66d8ba5e9ac\") " pod="openshift-dns/node-resolver-bflth"
Nov 02 01:24:25.147317 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146569    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-sys-fs\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:24:25.147317 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146597    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-cni-netd\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.147317 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146621    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ef8451f5-e54e-42e3-aad7-16e13fb70d81-ovnkube-config\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.147317 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146647    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwct4\" (UniqueName: \"kubernetes.io/projected/b6238ba5-f475-4e61-94d8-7d7bc13b2458-kube-api-access-zwct4\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:24:25.147317 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146685    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9256ae6-c79a-46f6-b6bd-8176b8907045-tmp\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.147317 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146742    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-etc-openvswitch\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.147317 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146761    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ef8451f5-e54e-42e3-aad7-16e13fb70d81-ovn-node-metrics-cert\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.147317 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146781    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrkz9\" (UniqueName: \"kubernetes.io/projected/d352c5c1-44a0-4929-a43a-30c723958ce7-kube-api-access-mrkz9\") pod \"node-ca-9lvcx\" (UID: \"d352c5c1-44a0-4929-a43a-30c723958ce7\") " pod="openshift-image-registry/node-ca-9lvcx"
Nov 02 01:24:25.147317 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146809    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-multus-conf-dir\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.147748 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146832    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-systemd\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.147748 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146846    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-host\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.147748 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146879    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsgjl\" (UniqueName: \"kubernetes.io/projected/c9256ae6-c79a-46f6-b6bd-8176b8907045-kube-api-access-nsgjl\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.147748 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146899    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9w59\" (UniqueName: \"kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59\") pod \"network-check-target-w4jf8\" (UID: \"4240f2d9-dd99-4ca9-8447-7706cebcfa08\") " pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:24:25.147748 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146915    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-kubelet-config\") pod \"global-pull-secret-syncer-9kqsh\" (UID: \"26cf53e3-9cf3-4ad5-afe2-5c241f6749ad\") " pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:25.147748 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146930    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret\") pod \"global-pull-secret-syncer-9kqsh\" (UID: \"26cf53e3-9cf3-4ad5-afe2-5c241f6749ad\") " pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:25.147748 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.146969    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-run-netns\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.147748 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.147003    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-node-log\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.147748 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.147035    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b0d87141-a1da-4d64-b01e-e66d8ba5e9ac-hosts-file\") pod \"node-resolver-bflth\" (UID: \"b0d87141-a1da-4d64-b01e-e66d8ba5e9ac\") " pod="openshift-dns/node-resolver-bflth"
Nov 02 01:24:25.147748 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.147056    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-cnibin\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.147748 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.147079    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-cni-binary-copy\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.147748 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.147094    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-run-netns\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.147748 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.147107    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-etc-kubernetes\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.147748 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.147120    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-run\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.147748 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.147135    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65dfab40-fb6d-46d7-98f1-8243e194c176-host-slash\") pod \"iptables-alerter-mjlgj\" (UID: \"65dfab40-fb6d-46d7-98f1-8243e194c176\") " pod="openshift-network-operator/iptables-alerter-mjlgj"
Nov 02 01:24:25.147748 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.147148    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6238ba5-f475-4e61-94d8-7d7bc13b2458-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:24:25.147748 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.147162    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-device-dir\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:24:25.148217 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.147190    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-var-lib-kubelet\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.148217 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.147223    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-cni-bin\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.148499 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.148480    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 01:24:25.148581 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.148482    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:24:25.148581 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.148545    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kv2jl\""
Nov 02 01:24:25.148581 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.148571    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 01:24:25.148581 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.148551    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 01:24:25.148751 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.148615    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 01:24:25.148751 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.148626    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 01:24:25.148751 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.148650    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 01:24:25.148751 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.148717    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 01:24:25.148751 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.148719    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pw4q8\""
Nov 02 01:24:25.148751 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.148745    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 01:24:25.149220 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.149205    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 01:24:25.149301 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.149210    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 01:24:25.149354 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.149316    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wwr4r\""
Nov 02 01:24:25.151329 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.151311    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 01:24:25.151424 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.151353    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 01:24:25.151424 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.151355    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 01:24:25.151424 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.151417    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2dvkb\""
Nov 02 01:24:25.153417 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.153400    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4gwsn\""
Nov 02 01:24:25.153504 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.153420    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 01:24:25.153504 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.153420    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 01:24:25.153504 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.153490    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 01:24:25.153596 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.153495    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 01:24:25.242538 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.242511    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 01:24:25.242664 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.242580    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4822p"
Nov 02 01:24:25.246353 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.246329    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 01:24:25.246353 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.246344    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 01:24:25.246556 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.246333    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 01:24:25.246556 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.246455    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:24:25.246556 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.246504    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 01:24:25.246556 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.246530    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 01:24:25.246556 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.246536    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 01:24:25.246556 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.246544    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p5zpq\""
Nov 02 01:24:25.246794 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.246536    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gh69v\""
Nov 02 01:24:25.246794 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.246588    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 01:24:25.246794 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.246719    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-gsg88\""
Nov 02 01:24:25.246794 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.246783    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 01:24:25.246912 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.246859    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wfbzc\""
Nov 02 01:24:25.247825 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.247809    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6238ba5-f475-4e61-94d8-7d7bc13b2458-system-cni-dir\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:24:25.247865 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.247833    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6238ba5-f475-4e61-94d8-7d7bc13b2458-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:24:25.247865 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.247850    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-run-k8s-cni-cncf-io\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.247949 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.247868    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-kubelet\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.247949 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.247884    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs\") pod \"network-metrics-daemon-rfwwn\" (UID: \"9eb605c8-f995-49ae-a1b8-06aa55160b13\") " pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:25.247949 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.247929    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6238ba5-f475-4e61-94d8-7d7bc13b2458-system-cni-dir\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:24:25.248099 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.247955    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-run-k8s-cni-cncf-io\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.248099 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.247970    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-kubelet\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.248099 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:25.247974    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:24:25.248242 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:25.248142    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs podName:9eb605c8-f995-49ae-a1b8-06aa55160b13 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:25.74810493 +0000 UTC m=+2.209622602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs") pod "network-metrics-daemon-rfwwn" (UID: "9eb605c8-f995-49ae-a1b8-06aa55160b13") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:24:25.248459 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.248441    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6238ba5-f475-4e61-94d8-7d7bc13b2458-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:24:25.248516 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.247975    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p47d\" (UniqueName: \"kubernetes.io/projected/7407afb5-5371-46c7-93ad-896d94a6f015-kube-api-access-9p47d\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:24:25.248568 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.248553    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-systemd-units\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.248620 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.248589    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-run-systemd\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.248667 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.248627    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-systemd-units\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.248667 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.248620    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6238ba5-f475-4e61-94d8-7d7bc13b2458-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:24:25.248667 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.248648    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-run-systemd\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.248817 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.248668    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d87141-a1da-4d64-b01e-e66d8ba5e9ac-tmp-dir\") pod \"node-resolver-bflth\" (UID: \"b0d87141-a1da-4d64-b01e-e66d8ba5e9ac\") " pod="openshift-dns/node-resolver-bflth"
Nov 02 01:24:25.248817 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.248701    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-sys-fs\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:24:25.248817 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.248725    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-cni-netd\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.248817 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.248748    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ef8451f5-e54e-42e3-aad7-16e13fb70d81-ovnkube-config\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.248817 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.248773    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwct4\" (UniqueName: \"kubernetes.io/projected/b6238ba5-f475-4e61-94d8-7d7bc13b2458-kube-api-access-zwct4\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:24:25.248817 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.248800    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-cni-netd\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.249052 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.248834    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-sys-fs\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:24:25.249052 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.248849    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9256ae6-c79a-46f6-b6bd-8176b8907045-tmp\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.249052 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.248887    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-etc-openvswitch\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.249052 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.248950    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d87141-a1da-4d64-b01e-e66d8ba5e9ac-tmp-dir\") pod \"node-resolver-bflth\" (UID: \"b0d87141-a1da-4d64-b01e-e66d8ba5e9ac\") " pod="openshift-dns/node-resolver-bflth"
Nov 02 01:24:25.249052 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.248951    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ef8451f5-e54e-42e3-aad7-16e13fb70d81-ovn-node-metrics-cert\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.249052 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.248985    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrkz9\" (UniqueName: \"kubernetes.io/projected/d352c5c1-44a0-4929-a43a-30c723958ce7-kube-api-access-mrkz9\") pod \"node-ca-9lvcx\" (UID: \"d352c5c1-44a0-4929-a43a-30c723958ce7\") " pod="openshift-image-registry/node-ca-9lvcx"
Nov 02 01:24:25.249052 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.249017    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-multus-conf-dir\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.249052 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.249044    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-etc-openvswitch\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.249422 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.249085    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-systemd\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.249422 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.249156    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-multus-conf-dir\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.249422 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.249176    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-systemd\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.249422 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.249197    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-host\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.249422 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.249253    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-host\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.249422 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.249254    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsgjl\" (UniqueName: \"kubernetes.io/projected/c9256ae6-c79a-46f6-b6bd-8176b8907045-kube-api-access-nsgjl\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.249422 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.249350    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9w59\" (UniqueName: \"kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59\") pod \"network-check-target-w4jf8\" (UID: \"4240f2d9-dd99-4ca9-8447-7706cebcfa08\") " pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:24:25.249422 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.249386    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6238ba5-f475-4e61-94d8-7d7bc13b2458-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:24:25.249422 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.249386    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-kubelet-config\") pod \"global-pull-secret-syncer-9kqsh\" (UID: \"26cf53e3-9cf3-4ad5-afe2-5c241f6749ad\") " pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:25.249768 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.249429    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret\") pod \"global-pull-secret-syncer-9kqsh\" (UID: \"26cf53e3-9cf3-4ad5-afe2-5c241f6749ad\") " pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:25.249768 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.249454    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-run-netns\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.249768 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.249471    2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 01:24:25.249768 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.249518    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-kubelet-config\") pod \"global-pull-secret-syncer-9kqsh\" (UID: \"26cf53e3-9cf3-4ad5-afe2-5c241f6749ad\") " pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:25.249768 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:25.249601    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:24:25.249768 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.249715    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ef8451f5-e54e-42e3-aad7-16e13fb70d81-ovnkube-config\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.249768 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.249742    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-run-netns\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.249953 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.249925    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-node-log\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.249983 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.249958    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-node-log\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.249983 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.249968    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b0d87141-a1da-4d64-b01e-e66d8ba5e9ac-hosts-file\") pod \"node-resolver-bflth\" (UID: \"b0d87141-a1da-4d64-b01e-e66d8ba5e9ac\") " pod="openshift-dns/node-resolver-bflth"
Nov 02 01:24:25.250041 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:25.249994    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret podName:26cf53e3-9cf3-4ad5-afe2-5c241f6749ad nodeName:}" failed. No retries permitted until 2026-11-02 01:24:25.749976175 +0000 UTC m=+2.211493831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret") pod "global-pull-secret-syncer-9kqsh" (UID: "26cf53e3-9cf3-4ad5-afe2-5c241f6749ad") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:24:25.250041 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250014    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b0d87141-a1da-4d64-b01e-e66d8ba5e9ac-hosts-file\") pod \"node-resolver-bflth\" (UID: \"b0d87141-a1da-4d64-b01e-e66d8ba5e9ac\") " pod="openshift-dns/node-resolver-bflth"
Nov 02 01:24:25.250041 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250030    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-cnibin\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.250141 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250050    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-cni-binary-copy\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.250141 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250064    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-run-netns\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.250141 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250086    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-etc-kubernetes\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.250141 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250104    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-cnibin\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.250141 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250121    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-run-netns\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.250337 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250180    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-run\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.250337 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250217    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-etc-kubernetes\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.250337 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250232    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65dfab40-fb6d-46d7-98f1-8243e194c176-host-slash\") pod \"iptables-alerter-mjlgj\" (UID: \"65dfab40-fb6d-46d7-98f1-8243e194c176\") " pod="openshift-network-operator/iptables-alerter-mjlgj"
Nov 02 01:24:25.250337 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250260    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6238ba5-f475-4e61-94d8-7d7bc13b2458-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:24:25.250337 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250304    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-device-dir\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:24:25.250337 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250329    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-var-lib-kubelet\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.250612 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250347    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-run\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.250612 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250360    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-cni-bin\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.250612 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250395    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65dfab40-fb6d-46d7-98f1-8243e194c176-host-slash\") pod \"iptables-alerter-mjlgj\" (UID: \"65dfab40-fb6d-46d7-98f1-8243e194c176\") " pod="openshift-network-operator/iptables-alerter-mjlgj"
Nov 02 01:24:25.250612 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250462    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-var-lib-kubelet\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.250612 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250404    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-device-dir\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:24:25.250612 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250562    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fswd\" (UniqueName: \"kubernetes.io/projected/b0d87141-a1da-4d64-b01e-e66d8ba5e9ac-kube-api-access-9fswd\") pod \"node-resolver-bflth\" (UID: \"b0d87141-a1da-4d64-b01e-e66d8ba5e9ac\") " pod="openshift-dns/node-resolver-bflth"
Nov 02 01:24:25.250876 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250664    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-cni-binary-copy\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.250876 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250700    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-sysconfig\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.250876 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250734    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-cni-bin\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.250876 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250782    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-sys\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.250876 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250813    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdxpz\" (UniqueName: \"kubernetes.io/projected/9eb605c8-f995-49ae-a1b8-06aa55160b13-kube-api-access-zdxpz\") pod \"network-metrics-daemon-rfwwn\" (UID: \"9eb605c8-f995-49ae-a1b8-06aa55160b13\") " pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:25.250876 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250815    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6238ba5-f475-4e61-94d8-7d7bc13b2458-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:24:25.250876 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250818    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-sys\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.250876 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250835    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-var-lib-cni-bin\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.250876 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250783    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-sysconfig\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.251343 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250895    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbldg\" (UniqueName: \"kubernetes.io/projected/65dfab40-fb6d-46d7-98f1-8243e194c176-kube-api-access-dbldg\") pod \"iptables-alerter-mjlgj\" (UID: \"65dfab40-fb6d-46d7-98f1-8243e194c176\") " pod="openshift-network-operator/iptables-alerter-mjlgj"
Nov 02 01:24:25.251343 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250911    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-var-lib-cni-bin\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.251343 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250940    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-var-lib-openvswitch\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.251343 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250964    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/929a4d2e-02fa-4d5f-ace0-0ad3d41614b7-konnectivity-ca\") pod \"konnectivity-agent-vlg9k\" (UID: \"929a4d2e-02fa-4d5f-ace0-0ad3d41614b7\") " pod="kube-system/konnectivity-agent-vlg9k"
Nov 02 01:24:25.251343 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.250982    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-lib-modules\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.251343 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251015    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-tuned\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.251343 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251080    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:24:25.251343 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251085    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-lib-modules\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.251343 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251011    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-var-lib-openvswitch\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.251343 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251128    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-etc-selinux\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:24:25.251343 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251153    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-run-openvswitch\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.251343 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251173    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:24:25.251343 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251192    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/929a4d2e-02fa-4d5f-ace0-0ad3d41614b7-agent-certs\") pod \"konnectivity-agent-vlg9k\" (UID: \"929a4d2e-02fa-4d5f-ace0-0ad3d41614b7\") " pod="kube-system/konnectivity-agent-vlg9k"
Nov 02 01:24:25.251343 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251220    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-run-openvswitch\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.251343 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251233    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-sysctl-d\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.251343 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251254    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-etc-selinux\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:24:25.251343 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251297    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-registration-dir\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:24:25.252336 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251326    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d352c5c1-44a0-4929-a43a-30c723958ce7-host\") pod \"node-ca-9lvcx\" (UID: \"d352c5c1-44a0-4929-a43a-30c723958ce7\") " pod="openshift-image-registry/node-ca-9lvcx"
Nov 02 01:24:25.252336 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251354    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-run-ovn\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.252336 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251385    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-run-ovn-kubernetes\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.252336 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251382    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-sysctl-d\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.252336 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251462    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6238ba5-f475-4e61-94d8-7d7bc13b2458-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:24:25.252336 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251503    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-system-cni-dir\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.252336 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251529    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-registration-dir\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:24:25.252336 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251531    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-multus-socket-dir-parent\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.252336 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251586    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-hostroot\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.252336 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251597    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-run-ovn\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.252336 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251682    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-system-cni-dir\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.252336 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251545    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d352c5c1-44a0-4929-a43a-30c723958ce7-host\") pod \"node-ca-9lvcx\" (UID: \"d352c5c1-44a0-4929-a43a-30c723958ce7\") " pod="openshift-image-registry/node-ca-9lvcx"
Nov 02 01:24:25.252336 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251739    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-multus-daemon-config\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.252336 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251769    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-multus-socket-dir-parent\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.252336 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251770    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-slash\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.252336 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251826    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-run-ovn-kubernetes\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.252336 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251829    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ef8451f5-e54e-42e3-aad7-16e13fb70d81-ovnkube-script-lib\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.252336 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.251993    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d352c5c1-44a0-4929-a43a-30c723958ce7-serviceca\") pod \"node-ca-9lvcx\" (UID: \"d352c5c1-44a0-4929-a43a-30c723958ce7\") " pod="openshift-image-registry/node-ca-9lvcx"
Nov 02 01:24:25.253397 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252043    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzgv6\" (UniqueName: \"kubernetes.io/projected/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-kube-api-access-xzgv6\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.253397 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252065    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-kubernetes\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.253397 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252109    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-dbus\") pod \"global-pull-secret-syncer-9kqsh\" (UID: \"26cf53e3-9cf3-4ad5-afe2-5c241f6749ad\") " pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:25.253397 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252141    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65dfab40-fb6d-46d7-98f1-8243e194c176-iptables-alerter-script\") pod \"iptables-alerter-mjlgj\" (UID: \"65dfab40-fb6d-46d7-98f1-8243e194c176\") " pod="openshift-network-operator/iptables-alerter-mjlgj"
Nov 02 01:24:25.253397 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252163    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-log-socket\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.253397 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252205    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6238ba5-f475-4e61-94d8-7d7bc13b2458-cnibin\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:24:25.253397 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252224    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6238ba5-f475-4e61-94d8-7d7bc13b2458-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:24:25.253397 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252228    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-multus-cni-dir\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.253397 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252326    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-socket-dir\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:24:25.253397 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252358    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.253397 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252390    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-multus-daemon-config\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.253397 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252412    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-run-multus-certs\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.253397 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252439    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-modprobe-d\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.253397 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252459    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-hostroot\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.253397 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252463    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-sysctl-conf\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.253397 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252460    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ef8451f5-e54e-42e3-aad7-16e13fb70d81-ovnkube-script-lib\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.253397 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252513    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-var-lib-kubelet\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.253886 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252569    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-var-lib-cni-multus\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.253886 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252570    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-slash\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.253886 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252609    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-kubernetes\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.253886 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252628    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-log-socket\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.253886 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252656    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-multus-cni-dir\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.253886 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252679    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-dbus\") pod \"global-pull-secret-syncer-9kqsh\" (UID: \"26cf53e3-9cf3-4ad5-afe2-5c241f6749ad\") " pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:25.253886 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252722    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6238ba5-f475-4e61-94d8-7d7bc13b2458-cnibin\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:24:25.253886 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252769    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-socket-dir\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:24:25.253886 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252804    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-modprobe-d\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.253886 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252848    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.253886 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252873    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-run-multus-certs\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.253886 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252930    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-var-lib-kubelet\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.253886 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252942    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-sysctl-conf\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.253886 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252966    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-var-lib-cni-multus\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.253886 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.252989    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj9pc\" (UniqueName: \"kubernetes.io/projected/ef8451f5-e54e-42e3-aad7-16e13fb70d81-kube-api-access-gj9pc\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.253886 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.253016    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ef8451f5-e54e-42e3-aad7-16e13fb70d81-env-overrides\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.253886 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.253130    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65dfab40-fb6d-46d7-98f1-8243e194c176-iptables-alerter-script\") pod \"iptables-alerter-mjlgj\" (UID: \"65dfab40-fb6d-46d7-98f1-8243e194c176\") " pod="openshift-network-operator/iptables-alerter-mjlgj"
Nov 02 01:24:25.253886 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.253167    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d352c5c1-44a0-4929-a43a-30c723958ce7-serviceca\") pod \"node-ca-9lvcx\" (UID: \"d352c5c1-44a0-4929-a43a-30c723958ce7\") " pod="openshift-image-registry/node-ca-9lvcx"
Nov 02 01:24:25.254436 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.254298    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ef8451f5-e54e-42e3-aad7-16e13fb70d81-env-overrides\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.254436 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.254379    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9256ae6-c79a-46f6-b6bd-8176b8907045-tmp\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.254514 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.254443    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-tuned\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.254555 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.254526    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ef8451f5-e54e-42e3-aad7-16e13fb70d81-ovn-node-metrics-cert\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.254625 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.254607    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/929a4d2e-02fa-4d5f-ace0-0ad3d41614b7-konnectivity-ca\") pod \"konnectivity-agent-vlg9k\" (UID: \"929a4d2e-02fa-4d5f-ace0-0ad3d41614b7\") " pod="kube-system/konnectivity-agent-vlg9k"
Nov 02 01:24:25.256263 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.256245    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/929a4d2e-02fa-4d5f-ace0-0ad3d41614b7-agent-certs\") pod \"konnectivity-agent-vlg9k\" (UID: \"929a4d2e-02fa-4d5f-ace0-0ad3d41614b7\") " pod="kube-system/konnectivity-agent-vlg9k"
Nov 02 01:24:25.307325 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:25.307290    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:24:25.307325 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:25.307325    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:24:25.307510 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:25.307341    2574 projected.go:196] Error preparing data for projected volume kube-api-access-p9w59 for pod openshift-network-diagnostics/network-check-target-w4jf8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:24:25.307510 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:25.307434    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59 podName:4240f2d9-dd99-4ca9-8447-7706cebcfa08 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:25.807411056 +0000 UTC m=+2.268928733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-p9w59" (UniqueName: "kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59") pod "network-check-target-w4jf8" (UID: "4240f2d9-dd99-4ca9-8447-7706cebcfa08") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:24:25.309526 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.309495    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p47d\" (UniqueName: \"kubernetes.io/projected/7407afb5-5371-46c7-93ad-896d94a6f015-kube-api-access-9p47d\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:24:25.309788 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.309762    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsgjl\" (UniqueName: \"kubernetes.io/projected/c9256ae6-c79a-46f6-b6bd-8176b8907045-kube-api-access-nsgjl\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.309970 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.309935    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwct4\" (UniqueName: \"kubernetes.io/projected/b6238ba5-f475-4e61-94d8-7d7bc13b2458-kube-api-access-zwct4\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:24:25.310039 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.309995    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrkz9\" (UniqueName: \"kubernetes.io/projected/d352c5c1-44a0-4929-a43a-30c723958ce7-kube-api-access-mrkz9\") pod \"node-ca-9lvcx\" (UID: \"d352c5c1-44a0-4929-a43a-30c723958ce7\") " pod="openshift-image-registry/node-ca-9lvcx"
Nov 02 01:24:25.311423 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.311395    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbldg\" (UniqueName: \"kubernetes.io/projected/65dfab40-fb6d-46d7-98f1-8243e194c176-kube-api-access-dbldg\") pod \"iptables-alerter-mjlgj\" (UID: \"65dfab40-fb6d-46d7-98f1-8243e194c176\") " pod="openshift-network-operator/iptables-alerter-mjlgj"
Nov 02 01:24:25.311423 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.311410    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fswd\" (UniqueName: \"kubernetes.io/projected/b0d87141-a1da-4d64-b01e-e66d8ba5e9ac-kube-api-access-9fswd\") pod \"node-resolver-bflth\" (UID: \"b0d87141-a1da-4d64-b01e-e66d8ba5e9ac\") " pod="openshift-dns/node-resolver-bflth"
Nov 02 01:24:25.311805 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.311782    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdxpz\" (UniqueName: \"kubernetes.io/projected/9eb605c8-f995-49ae-a1b8-06aa55160b13-kube-api-access-zdxpz\") pod \"network-metrics-daemon-rfwwn\" (UID: \"9eb605c8-f995-49ae-a1b8-06aa55160b13\") " pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:25.312654 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.312632    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzgv6\" (UniqueName: \"kubernetes.io/projected/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-kube-api-access-xzgv6\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.313402 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.313386    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj9pc\" (UniqueName: \"kubernetes.io/projected/ef8451f5-e54e-42e3-aad7-16e13fb70d81-kube-api-access-gj9pc\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.435592 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.435512    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mjlgj"
Nov 02 01:24:25.461101 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.461074    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bflth"
Nov 02 01:24:25.464231 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.464206    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:25.468539 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:25.468510    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65dfab40_fb6d_46d7_98f1_8243e194c176.slice/crio-c05d8dfb9ad698f6250033cc556920c94cf3ea8930b18ed9c755533bf77b3a81 WatchSource:0}: Error finding container c05d8dfb9ad698f6250033cc556920c94cf3ea8930b18ed9c755533bf77b3a81: Status 404 returned error can't find the container with id c05d8dfb9ad698f6250033cc556920c94cf3ea8930b18ed9c755533bf77b3a81
Nov 02 01:24:25.468799 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:25.468775    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b7d72c2c054e228c464e14811dfb58.slice/crio-687f4030ba6f638575c614f535becdcd181b697bcf2ad0a9ada0c1c0c4333ee7 WatchSource:0}: Error finding container 687f4030ba6f638575c614f535becdcd181b697bcf2ad0a9ada0c1c0c4333ee7: Status 404 returned error can't find the container with id 687f4030ba6f638575c614f535becdcd181b697bcf2ad0a9ada0c1c0c4333ee7
Nov 02 01:24:25.474544 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.474532    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:24:25.476372 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:25.476350    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d87141_a1da_4d64_b01e_e66d8ba5e9ac.slice/crio-f7d3e82589ac6e2d16b3a6849e0d70977502e6f899b82854f1d87e695ef7bda8 WatchSource:0}: Error finding container f7d3e82589ac6e2d16b3a6849e0d70977502e6f899b82854f1d87e695ef7bda8: Status 404 returned error can't find the container with id f7d3e82589ac6e2d16b3a6849e0d70977502e6f899b82854f1d87e695ef7bda8
Nov 02 01:24:25.477113 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.477096    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9lvcx"
Nov 02 01:24:25.477423 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:25.477404    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8451f5_e54e_42e3_aad7_16e13fb70d81.slice/crio-0031722cf46aa740692ee13182e73c8900b973f16aeaef1ea37c75b215e60f6a WatchSource:0}: Error finding container 0031722cf46aa740692ee13182e73c8900b973f16aeaef1ea37c75b215e60f6a: Status 404 returned error can't find the container with id 0031722cf46aa740692ee13182e73c8900b973f16aeaef1ea37c75b215e60f6a
Nov 02 01:24:25.498908 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:25.498883    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd352c5c1_44a0_4929_a43a_30c723958ce7.slice/crio-20030ecde1b45e8bf123fa19e0c000ce41e91989c2962d24ef6c8f33ef32b5e1 WatchSource:0}: Error finding container 20030ecde1b45e8bf123fa19e0c000ce41e91989c2962d24ef6c8f33ef32b5e1: Status 404 returned error can't find the container with id 20030ecde1b45e8bf123fa19e0c000ce41e91989c2962d24ef6c8f33ef32b5e1
Nov 02 01:24:25.499895 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:25.499868    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48935a76d1ff2f699968aa919b673e4d.slice/crio-577994457ac2d3a23c2897c7646e07c02095ff54922aaae64cd7cacbbfcbffb4 WatchSource:0}: Error finding container 577994457ac2d3a23c2897c7646e07c02095ff54922aaae64cd7cacbbfcbffb4: Status 404 returned error can't find the container with id 577994457ac2d3a23c2897c7646e07c02095ff54922aaae64cd7cacbbfcbffb4
Nov 02 01:24:25.502218 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.502123    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vlg9k"
Nov 02 01:24:25.504184 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.504165    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:24:25.511328 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.511260    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n5rzm"
Nov 02 01:24:25.511533 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:25.511514    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod929a4d2e_02fa_4d5f_ace0_0ad3d41614b7.slice/crio-e5882f07a5330240dcedf16c806e69d7d044aecb03a1c741b2c69a16a5001d8b WatchSource:0}: Error finding container e5882f07a5330240dcedf16c806e69d7d044aecb03a1c741b2c69a16a5001d8b: Status 404 returned error can't find the container with id e5882f07a5330240dcedf16c806e69d7d044aecb03a1c741b2c69a16a5001d8b
Nov 02 01:24:25.513627 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:25.513610    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6238ba5_f475_4e61_94d8_7d7bc13b2458.slice/crio-921831ec3fa5204f8af52f0badb8fb2253ac0b7f7bd17aae39a59242775a11f2 WatchSource:0}: Error finding container 921831ec3fa5204f8af52f0badb8fb2253ac0b7f7bd17aae39a59242775a11f2: Status 404 returned error can't find the container with id 921831ec3fa5204f8af52f0badb8fb2253ac0b7f7bd17aae39a59242775a11f2
Nov 02 01:24:25.520693 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:25.520669    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c62a0f0_aefd_44fc_95c1_b2938317ee0d.slice/crio-f3f2992af63af9d80f550a68409bda39cd79d1b39871d40123005d198bf3fdfc WatchSource:0}: Error finding container f3f2992af63af9d80f550a68409bda39cd79d1b39871d40123005d198bf3fdfc: Status 404 returned error can't find the container with id f3f2992af63af9d80f550a68409bda39cd79d1b39871d40123005d198bf3fdfc
Nov 02 01:24:25.527987 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.527970    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:24:25.535855 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.535831    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:24:25.536938 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:25.536914    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9256ae6_c79a_46f6_b6bd_8176b8907045.slice/crio-6ab47a4915254b3c13e0970943155b7d3d181402e0b2efd873e11734d958d22a WatchSource:0}: Error finding container 6ab47a4915254b3c13e0970943155b7d3d181402e0b2efd873e11734d958d22a: Status 404 returned error can't find the container with id 6ab47a4915254b3c13e0970943155b7d3d181402e0b2efd873e11734d958d22a
Nov 02 01:24:25.544334 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:24:25.544313    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7407afb5_5371_46c7_93ad_896d94a6f015.slice/crio-0225b89e95550a2d9c8208d8c842473121f6b61a4a17ca74d36c1be1cf61c5e2 WatchSource:0}: Error finding container 0225b89e95550a2d9c8208d8c842473121f6b61a4a17ca74d36c1be1cf61c5e2: Status 404 returned error can't find the container with id 0225b89e95550a2d9c8208d8c842473121f6b61a4a17ca74d36c1be1cf61c5e2
Nov 02 01:24:25.546249 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:25.546229    2574 kuberuntime_manager.go:1707] "Unhandled Error" err="container csi-driver start failed in pod aws-ebs-csi-driver-node-tt978_openshift-cluster-csi-drivers(7407afb5-5371-46c7-93ad-896d94a6f015): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Nov 02 01:24:25.547260 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:25.547238    2574 kuberuntime_manager.go:1707] "Unhandled Error" err="container csi-node-driver-registrar start failed in pod aws-ebs-csi-driver-node-tt978_openshift-cluster-csi-drivers(7407afb5-5371-46c7-93ad-896d94a6f015): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Nov 02 01:24:25.548300 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:25.548262    2574 kuberuntime_manager.go:1707] "Unhandled Error" err="container csi-liveness-probe start failed in pod aws-ebs-csi-driver-node-tt978_openshift-cluster-csi-drivers(7407afb5-5371-46c7-93ad-896d94a6f015): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Nov 02 01:24:25.548363 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:25.548320    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-driver\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"csi-liveness-probe\" with ErrImagePull: \"pull QPS exceeded\"]" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978" podUID="7407afb5-5371-46c7-93ad-896d94a6f015"
Nov 02 01:24:25.756590 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.756555    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs\") pod \"network-metrics-daemon-rfwwn\" (UID: \"9eb605c8-f995-49ae-a1b8-06aa55160b13\") " pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:25.756590 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.756608    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret\") pod \"global-pull-secret-syncer-9kqsh\" (UID: \"26cf53e3-9cf3-4ad5-afe2-5c241f6749ad\") " pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:25.756821 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:25.756718    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:24:25.756821 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:25.756732    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:24:25.756821 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:25.756782    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret podName:26cf53e3-9cf3-4ad5-afe2-5c241f6749ad nodeName:}" failed. No retries permitted until 2026-11-02 01:24:26.756765166 +0000 UTC m=+3.218282821 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret") pod "global-pull-secret-syncer-9kqsh" (UID: "26cf53e3-9cf3-4ad5-afe2-5c241f6749ad") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:24:25.756821 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:25.756799    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs podName:9eb605c8-f995-49ae-a1b8-06aa55160b13 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:26.75679153 +0000 UTC m=+3.218309186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs") pod "network-metrics-daemon-rfwwn" (UID: "9eb605c8-f995-49ae-a1b8-06aa55160b13") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:24:25.857356 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.857288    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9w59\" (UniqueName: \"kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59\") pod \"network-check-target-w4jf8\" (UID: \"4240f2d9-dd99-4ca9-8447-7706cebcfa08\") " pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:24:25.858017 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:25.857477    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:24:25.858017 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:25.857497    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:24:25.858017 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:25.857506    2574 projected.go:196] Error preparing data for projected volume kube-api-access-p9w59 for pod openshift-network-diagnostics/network-check-target-w4jf8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:24:25.858017 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:25.857567    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59 podName:4240f2d9-dd99-4ca9-8447-7706cebcfa08 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:26.857552804 +0000 UTC m=+3.319070460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-p9w59" (UniqueName: "kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59") pod "network-check-target-w4jf8" (UID: "4240f2d9-dd99-4ca9-8447-7706cebcfa08") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:24:25.972474 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.972366    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978" event={"ID":"7407afb5-5371-46c7-93ad-896d94a6f015","Type":"ContainerStarted","Data":"0225b89e95550a2d9c8208d8c842473121f6b61a4a17ca74d36c1be1cf61c5e2"}
Nov 02 01:24:25.973711 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.973686    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n5rzm" event={"ID":"4c62a0f0-aefd-44fc-95c1-b2938317ee0d","Type":"ContainerStarted","Data":"f3f2992af63af9d80f550a68409bda39cd79d1b39871d40123005d198bf3fdfc"}
Nov 02 01:24:25.975603 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.975573    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vlg9k" event={"ID":"929a4d2e-02fa-4d5f-ace0-0ad3d41614b7","Type":"ContainerStarted","Data":"e5882f07a5330240dcedf16c806e69d7d044aecb03a1c741b2c69a16a5001d8b"}
Nov 02 01:24:25.976730 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.976703    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal" event={"ID":"48935a76d1ff2f699968aa919b673e4d","Type":"ContainerStarted","Data":"577994457ac2d3a23c2897c7646e07c02095ff54922aaae64cd7cacbbfcbffb4"}
Nov 02 01:24:25.976730 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:25.976709    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-driver\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:007ba215236827a40e1b33f3799d289065305ae886c05e51214d74d0ff24d62d\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a5e96bc702bc529b424e6c80f95d7d4fc4fe92f83df4a77c3b47eb3a30fe729\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"csi-liveness-probe\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4db2cbbaecb77e6b3a4e8dd7ee7a4d4fa9ab69787e4330711a3abb1277af2ac\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978" podUID="7407afb5-5371-46c7-93ad-896d94a6f015"
Nov 02 01:24:25.977799 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.977766    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9lvcx" event={"ID":"d352c5c1-44a0-4929-a43a-30c723958ce7","Type":"ContainerStarted","Data":"20030ecde1b45e8bf123fa19e0c000ce41e91989c2962d24ef6c8f33ef32b5e1"}
Nov 02 01:24:25.978890 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.978871    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bflth" event={"ID":"b0d87141-a1da-4d64-b01e-e66d8ba5e9ac","Type":"ContainerStarted","Data":"f7d3e82589ac6e2d16b3a6849e0d70977502e6f899b82854f1d87e695ef7bda8"}
Nov 02 01:24:25.980001 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.979982    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-stb4n" event={"ID":"c9256ae6-c79a-46f6-b6bd-8176b8907045","Type":"ContainerStarted","Data":"6ab47a4915254b3c13e0970943155b7d3d181402e0b2efd873e11734d958d22a"}
Nov 02 01:24:25.981148 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.981130    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmzfl" event={"ID":"b6238ba5-f475-4e61-94d8-7d7bc13b2458","Type":"ContainerStarted","Data":"921831ec3fa5204f8af52f0badb8fb2253ac0b7f7bd17aae39a59242775a11f2"}
Nov 02 01:24:25.982239 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.982221    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9" event={"ID":"ef8451f5-e54e-42e3-aad7-16e13fb70d81","Type":"ContainerStarted","Data":"0031722cf46aa740692ee13182e73c8900b973f16aeaef1ea37c75b215e60f6a"}
Nov 02 01:24:25.983198 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.983175    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal" event={"ID":"14b7d72c2c054e228c464e14811dfb58","Type":"ContainerStarted","Data":"687f4030ba6f638575c614f535becdcd181b697bcf2ad0a9ada0c1c0c4333ee7"}
Nov 02 01:24:25.984184 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:25.984166    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mjlgj" event={"ID":"65dfab40-fb6d-46d7-98f1-8243e194c176","Type":"ContainerStarted","Data":"c05d8dfb9ad698f6250033cc556920c94cf3ea8930b18ed9c755533bf77b3a81"}
Nov 02 01:24:26.189103 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:26.189071    2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:24:26.246139 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:26.246012    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 01:19:25 +0000 UTC" deadline="2028-03-29 15:06:06.513044954 +0000 UTC"
Nov 02 01:24:26.246139 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:26.246075    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12325h41m40.266972954s"
Nov 02 01:24:26.331131 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:26.331099    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:24:26.765368 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:26.765294    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs\") pod \"network-metrics-daemon-rfwwn\" (UID: \"9eb605c8-f995-49ae-a1b8-06aa55160b13\") " pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:26.765578 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:26.765395    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret\") pod \"global-pull-secret-syncer-9kqsh\" (UID: \"26cf53e3-9cf3-4ad5-afe2-5c241f6749ad\") " pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:26.765578 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:26.765564    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:24:26.765711 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:26.765634    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret podName:26cf53e3-9cf3-4ad5-afe2-5c241f6749ad nodeName:}" failed. No retries permitted until 2026-11-02 01:24:28.765614147 +0000 UTC m=+5.227131809 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret") pod "global-pull-secret-syncer-9kqsh" (UID: "26cf53e3-9cf3-4ad5-afe2-5c241f6749ad") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:24:26.765711 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:26.765705    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:24:26.765835 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:26.765740    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs podName:9eb605c8-f995-49ae-a1b8-06aa55160b13 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:28.765726307 +0000 UTC m=+5.227243965 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs") pod "network-metrics-daemon-rfwwn" (UID: "9eb605c8-f995-49ae-a1b8-06aa55160b13") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:24:26.866435 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:26.866394    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9w59\" (UniqueName: \"kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59\") pod \"network-check-target-w4jf8\" (UID: \"4240f2d9-dd99-4ca9-8447-7706cebcfa08\") " pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:24:26.866961 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:26.866615    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:24:26.866961 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:26.866636    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:24:26.866961 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:26.866652    2574 projected.go:196] Error preparing data for projected volume kube-api-access-p9w59 for pod openshift-network-diagnostics/network-check-target-w4jf8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:24:26.866961 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:26.866727    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59 podName:4240f2d9-dd99-4ca9-8447-7706cebcfa08 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:28.866707511 +0000 UTC m=+5.328225173 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-p9w59" (UniqueName: "kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59") pod "network-check-target-w4jf8" (UID: "4240f2d9-dd99-4ca9-8447-7706cebcfa08") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:24:26.967940 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:26.967900    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:26.968150 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:26.968065    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kqsh" podUID="26cf53e3-9cf3-4ad5-afe2-5c241f6749ad"
Nov 02 01:24:26.968218 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:26.968161    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:26.968277 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:26.968247    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfwwn" podUID="9eb605c8-f995-49ae-a1b8-06aa55160b13"
Nov 02 01:24:26.968380 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:26.968362    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:24:26.968469 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:26.968449    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4jf8" podUID="4240f2d9-dd99-4ca9-8447-7706cebcfa08"
Nov 02 01:24:27.246865 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:27.246763    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 01:19:25 +0000 UTC" deadline="2028-06-28 12:31:24.338876214 +0000 UTC"
Nov 02 01:24:27.246865 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:27.246812    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14507h6m57.092069297s"
Nov 02 01:24:28.782582 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:28.782534    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret\") pod \"global-pull-secret-syncer-9kqsh\" (UID: \"26cf53e3-9cf3-4ad5-afe2-5c241f6749ad\") " pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:28.783246 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:28.782619    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs\") pod \"network-metrics-daemon-rfwwn\" (UID: \"9eb605c8-f995-49ae-a1b8-06aa55160b13\") " pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:28.783246 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:28.782760    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:24:28.783246 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:28.782827    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs podName:9eb605c8-f995-49ae-a1b8-06aa55160b13 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:32.782808349 +0000 UTC m=+9.244326023 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs") pod "network-metrics-daemon-rfwwn" (UID: "9eb605c8-f995-49ae-a1b8-06aa55160b13") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:24:28.783246 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:28.782891    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:24:28.783246 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:28.782924    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret podName:26cf53e3-9cf3-4ad5-afe2-5c241f6749ad nodeName:}" failed. No retries permitted until 2026-11-02 01:24:32.782912206 +0000 UTC m=+9.244429877 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret") pod "global-pull-secret-syncer-9kqsh" (UID: "26cf53e3-9cf3-4ad5-afe2-5c241f6749ad") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:24:28.883766 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:28.883725    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9w59\" (UniqueName: \"kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59\") pod \"network-check-target-w4jf8\" (UID: \"4240f2d9-dd99-4ca9-8447-7706cebcfa08\") " pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:24:28.883950 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:28.883927    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:24:28.883950 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:28.883949    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:24:28.884089 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:28.883963    2574 projected.go:196] Error preparing data for projected volume kube-api-access-p9w59 for pod openshift-network-diagnostics/network-check-target-w4jf8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:24:28.884089 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:28.884027    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59 podName:4240f2d9-dd99-4ca9-8447-7706cebcfa08 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:32.884008945 +0000 UTC m=+9.345526605 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-p9w59" (UniqueName: "kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59") pod "network-check-target-w4jf8" (UID: "4240f2d9-dd99-4ca9-8447-7706cebcfa08") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:24:28.968569 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:28.968526    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:28.968736 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:28.968687    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kqsh" podUID="26cf53e3-9cf3-4ad5-afe2-5c241f6749ad"
Nov 02 01:24:28.968789 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:28.968762    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:28.968866 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:28.968843    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfwwn" podUID="9eb605c8-f995-49ae-a1b8-06aa55160b13"
Nov 02 01:24:28.968913 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:28.968893    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:24:28.968974 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:28.968956    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4jf8" podUID="4240f2d9-dd99-4ca9-8447-7706cebcfa08"
Nov 02 01:24:30.011254 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:30.010405    2574 generic.go:356] "Generic (PLEG): container finished" podID="b6238ba5-f475-4e61-94d8-7d7bc13b2458" containerID="0dcb69b56eb122074d5a6c4212ae6bb997769ee7849913be25f474d2dc5f4f46" exitCode=0
Nov 02 01:24:30.011254 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:30.010739    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmzfl" event={"ID":"b6238ba5-f475-4e61-94d8-7d7bc13b2458","Type":"ContainerDied","Data":"0dcb69b56eb122074d5a6c4212ae6bb997769ee7849913be25f474d2dc5f4f46"}
Nov 02 01:24:30.969226 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:30.968636    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:30.969226 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:30.968783    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kqsh" podUID="26cf53e3-9cf3-4ad5-afe2-5c241f6749ad"
Nov 02 01:24:30.969226 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:30.968879    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:30.969226 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:30.968971    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfwwn" podUID="9eb605c8-f995-49ae-a1b8-06aa55160b13"
Nov 02 01:24:30.969226 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:30.969034    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:24:30.969226 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:30.969129    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4jf8" podUID="4240f2d9-dd99-4ca9-8447-7706cebcfa08"
Nov 02 01:24:31.019586 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:31.018933    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal" event={"ID":"14b7d72c2c054e228c464e14811dfb58","Type":"ContainerStarted","Data":"5f1ef6b25227e3114be00aa9226a89f4ad7c50eb2abca5c1d5064396520fdcb0"}
Nov 02 01:24:31.022415 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:31.021808    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vlg9k" event={"ID":"929a4d2e-02fa-4d5f-ace0-0ad3d41614b7","Type":"ContainerStarted","Data":"fb8e72b85cc59168f44d13371220c65c0442ff7f23cd53f427974b7cb7eb75a9"}
Nov 02 01:24:31.026431 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:31.025778    2574 generic.go:356] "Generic (PLEG): container finished" podID="48935a76d1ff2f699968aa919b673e4d" containerID="effe724ff6b15d0a421751ff5e6d416b9676a987681b1d0c044279c649ece3be" exitCode=0
Nov 02 01:24:31.026431 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:31.025825    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal" event={"ID":"48935a76d1ff2f699968aa919b673e4d","Type":"ContainerDied","Data":"effe724ff6b15d0a421751ff5e6d416b9676a987681b1d0c044279c649ece3be"}
Nov 02 01:24:31.222314 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:31.222268    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vlg9k"
Nov 02 01:24:31.222505 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:31.222410    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vlg9k"
Nov 02 01:24:31.269556 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:31.269487    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal" podStartSLOduration=6.269466164 podStartE2EDuration="6.269466164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:31.142547397 +0000 UTC m=+7.604065078" watchObservedRunningTime="2026-11-02 01:24:31.269466164 +0000 UTC m=+7.730983847"
Nov 02 01:24:31.521920 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:31.521807    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vlg9k" podStartSLOduration=2.767097327 podStartE2EDuration="7.521782801s" totalImagesPullingTime="4.754685474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:24:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:31.394967641 +0000 UTC m=+7.856485320" watchObservedRunningTime="2026-11-02 01:24:31.521782801 +0000 UTC m=+7.983300480"
Nov 02 01:24:32.036571 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:32.036519    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal" event={"ID":"48935a76d1ff2f699968aa919b673e4d","Type":"ContainerStarted","Data":"b0112150ec9f8bf128d80e5b7d75dd2683457017c04e8a2e5d6a9db82ba58419"}
Nov 02 01:24:32.041918 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:32.041885    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9lvcx" event={"ID":"d352c5c1-44a0-4929-a43a-30c723958ce7","Type":"ContainerStarted","Data":"a1d3f8e743dca577440b6cf87f35fe643085a332856d00e9c68a80e01209ee58"}
Nov 02 01:24:32.042491 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:32.042430    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vlg9k"
Nov 02 01:24:32.043334 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:32.043274    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vlg9k"
Nov 02 01:24:32.162374 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:32.161503    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal" podStartSLOduration=7.161482458 podStartE2EDuration="7.161482458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:32.16107235 +0000 UTC m=+8.622590029" watchObservedRunningTime="2026-11-02 01:24:32.161482458 +0000 UTC m=+8.623000137"
Nov 02 01:24:32.410494 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:32.410163    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9lvcx" podStartSLOduration=2.3521329140000002 podStartE2EDuration="8.410141167s" totalImagesPullingTime="6.058008253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:32.285840089 +0000 UTC m=+8.747357768" watchObservedRunningTime="2026-11-02 01:24:32.410141167 +0000 UTC m=+8.871658878"
Nov 02 01:24:32.815969 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:32.815876    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs\") pod \"network-metrics-daemon-rfwwn\" (UID: \"9eb605c8-f995-49ae-a1b8-06aa55160b13\") " pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:32.815969 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:32.815962    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret\") pod \"global-pull-secret-syncer-9kqsh\" (UID: \"26cf53e3-9cf3-4ad5-afe2-5c241f6749ad\") " pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:32.816219 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:32.816153    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:24:32.816279 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:32.816225    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret podName:26cf53e3-9cf3-4ad5-afe2-5c241f6749ad nodeName:}" failed. No retries permitted until 2026-11-02 01:24:40.816205143 +0000 UTC m=+17.277722816 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret") pod "global-pull-secret-syncer-9kqsh" (UID: "26cf53e3-9cf3-4ad5-afe2-5c241f6749ad") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:24:32.816395 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:32.816301    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:24:32.816395 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:32.816366    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs podName:9eb605c8-f995-49ae-a1b8-06aa55160b13 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:40.816344308 +0000 UTC m=+17.277861965 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs") pod "network-metrics-daemon-rfwwn" (UID: "9eb605c8-f995-49ae-a1b8-06aa55160b13") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:24:32.916337 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:32.916285    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9w59\" (UniqueName: \"kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59\") pod \"network-check-target-w4jf8\" (UID: \"4240f2d9-dd99-4ca9-8447-7706cebcfa08\") " pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:24:32.916533 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:32.916515    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:24:32.916584 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:32.916541    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:24:32.916584 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:32.916556    2574 projected.go:196] Error preparing data for projected volume kube-api-access-p9w59 for pod openshift-network-diagnostics/network-check-target-w4jf8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:24:32.916653 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:32.916613    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59 podName:4240f2d9-dd99-4ca9-8447-7706cebcfa08 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:40.916594538 +0000 UTC m=+17.378112194 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-p9w59" (UniqueName: "kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59") pod "network-check-target-w4jf8" (UID: "4240f2d9-dd99-4ca9-8447-7706cebcfa08") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:24:32.968049 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:32.968005    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:32.968227 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:32.968190    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfwwn" podUID="9eb605c8-f995-49ae-a1b8-06aa55160b13"
Nov 02 01:24:32.968277 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:32.968268    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:24:32.968390 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:32.968361    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4jf8" podUID="4240f2d9-dd99-4ca9-8447-7706cebcfa08"
Nov 02 01:24:32.968479 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:32.968463    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:32.968621 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:32.968597    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kqsh" podUID="26cf53e3-9cf3-4ad5-afe2-5c241f6749ad"
Nov 02 01:24:34.053737 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:34.053696    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n5rzm" event={"ID":"4c62a0f0-aefd-44fc-95c1-b2938317ee0d","Type":"ContainerStarted","Data":"0a2100c23f61f39b3b093cbf1d5893e2f8255ce81a35a7a8659fd3c401da3e1e"}
Nov 02 01:24:34.185238 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:34.185178    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-n5rzm" podStartSLOduration=1.758223624 podStartE2EDuration="10.185162628s" totalImagesPullingTime="8.426939004s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:34.185009326 +0000 UTC m=+10.646527007" watchObservedRunningTime="2026-11-02 01:24:34.185162628 +0000 UTC m=+10.646680306"
Nov 02 01:24:34.968072 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:34.968047    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:34.968151 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:34.968072    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:34.968254 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:34.968207    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kqsh" podUID="26cf53e3-9cf3-4ad5-afe2-5c241f6749ad"
Nov 02 01:24:34.968254 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:34.968255    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:24:34.968479 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:34.968397    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfwwn" podUID="9eb605c8-f995-49ae-a1b8-06aa55160b13"
Nov 02 01:24:34.968609 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:34.968488    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4jf8" podUID="4240f2d9-dd99-4ca9-8447-7706cebcfa08"
Nov 02 01:24:35.058217 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:35.058165    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bflth" event={"ID":"b0d87141-a1da-4d64-b01e-e66d8ba5e9ac","Type":"ContainerStarted","Data":"4406b89b3f70de45effffeaa80d90e998f7a104d4415c89ea6f190851ee84128"}
Nov 02 01:24:35.183429 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:35.183368    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-bflth" podStartSLOduration=2.275194162 podStartE2EDuration="11.183323151s" totalImagesPullingTime="8.908128989s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:35.183058527 +0000 UTC m=+11.644576210" watchObservedRunningTime="2026-11-02 01:24:35.183323151 +0000 UTC m=+11.644840833"
Nov 02 01:24:36.062164 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:36.062127    2574 generic.go:356] "Generic (PLEG): container finished" podID="b6238ba5-f475-4e61-94d8-7d7bc13b2458" containerID="d755b7d37b846d9ff794bf9ada723a11092bacd52ff0900c4b03d1af21c94a27" exitCode=0
Nov 02 01:24:36.062648 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:36.062261    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmzfl" event={"ID":"b6238ba5-f475-4e61-94d8-7d7bc13b2458","Type":"ContainerDied","Data":"d755b7d37b846d9ff794bf9ada723a11092bacd52ff0900c4b03d1af21c94a27"}
Nov 02 01:24:36.968050 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:36.967810    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:36.968228 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:36.967887    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:24:36.968309 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:36.967997    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:36.968309 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:36.968141    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kqsh" podUID="26cf53e3-9cf3-4ad5-afe2-5c241f6749ad"
Nov 02 01:24:36.968309 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:36.968267    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4jf8" podUID="4240f2d9-dd99-4ca9-8447-7706cebcfa08"
Nov 02 01:24:36.968425 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:36.968345    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfwwn" podUID="9eb605c8-f995-49ae-a1b8-06aa55160b13"
Nov 02 01:24:37.066866 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:37.066777    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-stb4n" event={"ID":"c9256ae6-c79a-46f6-b6bd-8176b8907045","Type":"ContainerStarted","Data":"d8f4125ef4ac6183d0bde28e223ae2293644eb4a097446d972052bceaf4d7878"}
Nov 02 01:24:37.068547 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:37.068516    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mjlgj" event={"ID":"65dfab40-fb6d-46d7-98f1-8243e194c176","Type":"ContainerStarted","Data":"408975653f1fd9793eb7b39a8425b2daea7885eb7e6cc15630b65919e206f1b7"}
Nov 02 01:24:37.207192 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:37.207095    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-stb4n" podStartSLOduration=2.360523599 podStartE2EDuration="13.207077854s" totalImagesPullingTime="10.846554255s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:37.206559481 +0000 UTC m=+13.668077157" watchObservedRunningTime="2026-11-02 01:24:37.207077854 +0000 UTC m=+13.668595526"
Nov 02 01:24:37.322430 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:37.322322    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mjlgj" podStartSLOduration=3.72588316 podStartE2EDuration="13.322295054s" totalImagesPullingTime="9.596411894s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:37.321451704 +0000 UTC m=+13.782969398" watchObservedRunningTime="2026-11-02 01:24:37.322295054 +0000 UTC m=+13.783812733"
Nov 02 01:24:38.073591 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:38.073551    2574 generic.go:356] "Generic (PLEG): container finished" podID="b6238ba5-f475-4e61-94d8-7d7bc13b2458" containerID="b7197164ea85fd59c26fc348829cc5f8aab3b8b46960bf850ce86902cac955b8" exitCode=0
Nov 02 01:24:38.074068 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:38.073838    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmzfl" event={"ID":"b6238ba5-f475-4e61-94d8-7d7bc13b2458","Type":"ContainerDied","Data":"b7197164ea85fd59c26fc348829cc5f8aab3b8b46960bf850ce86902cac955b8"}
Nov 02 01:24:38.968523 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:38.968495    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:38.968693 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:38.968602    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfwwn" podUID="9eb605c8-f995-49ae-a1b8-06aa55160b13"
Nov 02 01:24:38.968986 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:38.968960    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:24:38.969072 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:38.969035    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4jf8" podUID="4240f2d9-dd99-4ca9-8447-7706cebcfa08"
Nov 02 01:24:38.969124 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:38.969072    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:38.969167 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:38.969132    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kqsh" podUID="26cf53e3-9cf3-4ad5-afe2-5c241f6749ad"
Nov 02 01:24:40.079365 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:40.079326    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978" event={"ID":"7407afb5-5371-46c7-93ad-896d94a6f015","Type":"ContainerStarted","Data":"f78b4f12aa88a07cbc39a35671d3b9649b1c4fc1f48ffe647cd9e1268eadeda1"}
Nov 02 01:24:40.082388 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:40.082357    2574 generic.go:356] "Generic (PLEG): container finished" podID="b6238ba5-f475-4e61-94d8-7d7bc13b2458" containerID="ea00fa33606ed3cbc2ae2cf2fe3b983d0216f0b3dd023254ec146cffb8598534" exitCode=0
Nov 02 01:24:40.082520 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:40.082398    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmzfl" event={"ID":"b6238ba5-f475-4e61-94d8-7d7bc13b2458","Type":"ContainerDied","Data":"ea00fa33606ed3cbc2ae2cf2fe3b983d0216f0b3dd023254ec146cffb8598534"}
Nov 02 01:24:40.877815 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:40.877776    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs\") pod \"network-metrics-daemon-rfwwn\" (UID: \"9eb605c8-f995-49ae-a1b8-06aa55160b13\") " pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:40.877980 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:40.877829    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret\") pod \"global-pull-secret-syncer-9kqsh\" (UID: \"26cf53e3-9cf3-4ad5-afe2-5c241f6749ad\") " pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:40.877980 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:40.877927    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:24:40.877980 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:40.877929    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:24:40.877980 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:40.877978    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret podName:26cf53e3-9cf3-4ad5-afe2-5c241f6749ad nodeName:}" failed. No retries permitted until 2026-11-02 01:24:56.877966203 +0000 UTC m=+33.339483857 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret") pod "global-pull-secret-syncer-9kqsh" (UID: "26cf53e3-9cf3-4ad5-afe2-5c241f6749ad") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:24:40.878126 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:40.877993    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs podName:9eb605c8-f995-49ae-a1b8-06aa55160b13 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:56.877986048 +0000 UTC m=+33.339503701 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs") pod "network-metrics-daemon-rfwwn" (UID: "9eb605c8-f995-49ae-a1b8-06aa55160b13") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:24:40.968455 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:40.968420    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:40.968654 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:40.968430    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:40.968654 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:40.968536    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfwwn" podUID="9eb605c8-f995-49ae-a1b8-06aa55160b13"
Nov 02 01:24:40.968654 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:40.968442    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:24:40.968654 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:40.968585    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kqsh" podUID="26cf53e3-9cf3-4ad5-afe2-5c241f6749ad"
Nov 02 01:24:40.968794 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:40.968655    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4jf8" podUID="4240f2d9-dd99-4ca9-8447-7706cebcfa08"
Nov 02 01:24:40.978600 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:40.978581    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9w59\" (UniqueName: \"kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59\") pod \"network-check-target-w4jf8\" (UID: \"4240f2d9-dd99-4ca9-8447-7706cebcfa08\") " pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:24:40.978718 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:40.978705    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:24:40.978757 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:40.978721    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:24:40.978757 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:40.978731    2574 projected.go:196] Error preparing data for projected volume kube-api-access-p9w59 for pod openshift-network-diagnostics/network-check-target-w4jf8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:24:40.978818 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:40.978770    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59 podName:4240f2d9-dd99-4ca9-8447-7706cebcfa08 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:56.978758628 +0000 UTC m=+33.440276281 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-p9w59" (UniqueName: "kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59") pod "network-check-target-w4jf8" (UID: "4240f2d9-dd99-4ca9-8447-7706cebcfa08") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:24:41.464352 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:41.464170    2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 01:24:41.921551 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:41.921439    2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T01:24:41.464333403Z","UUID":"5c2f7e2a-2954-4c92-8fdb-3d38e9070b35","Handler":null,"Name":"","Endpoint":""}
Nov 02 01:24:41.923126 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:41.923101    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 01:24:41.923126 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:41.923133    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 01:24:42.090426 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:42.090390    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9" event={"ID":"ef8451f5-e54e-42e3-aad7-16e13fb70d81","Type":"ContainerStarted","Data":"a4f9a38dbf1d235f0d77338d4f4d21cc1c49b1eea684db5065ded6bbde1bade2"}
Nov 02 01:24:42.090630 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:42.090489    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9" event={"ID":"ef8451f5-e54e-42e3-aad7-16e13fb70d81","Type":"ContainerStarted","Data":"3f6328a551e02369c0b1863260189e6344588824522f0d71847607c88fe0ad29"}
Nov 02 01:24:42.090630 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:42.090542    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9" event={"ID":"ef8451f5-e54e-42e3-aad7-16e13fb70d81","Type":"ContainerStarted","Data":"2bd8cd29215c5af6b8e3c59d5bb4c0ff7a253a69f2d8a3261e8b8807aa0e3ade"}
Nov 02 01:24:42.090630 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:42.090575    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9" event={"ID":"ef8451f5-e54e-42e3-aad7-16e13fb70d81","Type":"ContainerStarted","Data":"74f626c8f982cd7badc9a6d7f121e13d11864f58545ff00ec46604be0579c99a"}
Nov 02 01:24:42.090630 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:42.090612    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9" event={"ID":"ef8451f5-e54e-42e3-aad7-16e13fb70d81","Type":"ContainerStarted","Data":"90ebb0dbefd95f9fac5fbd2981987030a66bfd23b0198acb9828235f767f1a1c"}
Nov 02 01:24:42.090833 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:42.090639    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9" event={"ID":"ef8451f5-e54e-42e3-aad7-16e13fb70d81","Type":"ContainerStarted","Data":"9f715e3e0123874d920039993da3baa62013bc054007c27a7e4502b86deeb348"}
Nov 02 01:24:42.092434 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:42.092406    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978" event={"ID":"7407afb5-5371-46c7-93ad-896d94a6f015","Type":"ContainerStarted","Data":"047561549d53ff313213a62cb37c0ef863192891d05b7a476d0445989f61cb41"}
Nov 02 01:24:42.968235 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:42.968051    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:42.968648 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:42.968114    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:24:42.968648 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:42.968121    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:42.968648 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:42.968367    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kqsh" podUID="26cf53e3-9cf3-4ad5-afe2-5c241f6749ad"
Nov 02 01:24:42.968648 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:42.968450    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfwwn" podUID="9eb605c8-f995-49ae-a1b8-06aa55160b13"
Nov 02 01:24:42.968648 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:42.968511    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4jf8" podUID="4240f2d9-dd99-4ca9-8447-7706cebcfa08"
Nov 02 01:24:43.097049 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:43.097004    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978" event={"ID":"7407afb5-5371-46c7-93ad-896d94a6f015","Type":"ContainerStarted","Data":"901a330169972916a318fe9e4ad7c0133940b12ab5d32e795ac32a5c797862af"}
Nov 02 01:24:43.099627 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:43.099600    2574 generic.go:356] "Generic (PLEG): container finished" podID="b6238ba5-f475-4e61-94d8-7d7bc13b2458" containerID="8e778cb3addd874bc41c01983d7f8fd9a3799a2eeddf46b392e26720f5b5bc83" exitCode=0
Nov 02 01:24:43.099786 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:43.099638    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmzfl" event={"ID":"b6238ba5-f475-4e61-94d8-7d7bc13b2458","Type":"ContainerDied","Data":"8e778cb3addd874bc41c01983d7f8fd9a3799a2eeddf46b392e26720f5b5bc83"}
Nov 02 01:24:43.214175 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:43.214115    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978" podStartSLOduration=0.881005104 podStartE2EDuration="18.214096011s" totalImagesPullingTime="17.333090907s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=3 observedRunningTime="2026-11-02 01:24:43.213924088 +0000 UTC m=+19.675441763" watchObservedRunningTime="2026-11-02 01:24:43.214096011 +0000 UTC m=+19.675613687"
Nov 02 01:24:43.887972 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:43.887941    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bflth_b0d87141-a1da-4d64-b01e-e66d8ba5e9ac/dns-node-resolver/0.log"
Nov 02 01:24:43.888237 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:43.888213    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vlg9k_929a4d2e-02fa-4d5f-ace0-0ad3d41614b7/konnectivity-agent/0.log" limit=1024
Nov 02 01:24:44.087307 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:44.087261    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vlg9k_929a4d2e-02fa-4d5f-ace0-0ad3d41614b7/konnectivity-agent/0.log" limit=1024
Nov 02 01:24:44.104543 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:44.104508    2574 generic.go:356] "Generic (PLEG): container finished" podID="b6238ba5-f475-4e61-94d8-7d7bc13b2458" containerID="90b1f9961e8bc4581df246f8480beea1795285880385d6c153bbc71718348b11" exitCode=0
Nov 02 01:24:44.104693 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:44.104597    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmzfl" event={"ID":"b6238ba5-f475-4e61-94d8-7d7bc13b2458","Type":"ContainerDied","Data":"90b1f9961e8bc4581df246f8480beea1795285880385d6c153bbc71718348b11"}
Nov 02 01:24:44.107973 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:44.107946    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9" event={"ID":"ef8451f5-e54e-42e3-aad7-16e13fb70d81","Type":"ContainerStarted","Data":"e7df8b65845c2e022658943a235c56f3cba4de70cd31b5673cf971bdc1ffd141"}
Nov 02 01:24:44.343174 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:44.343141    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vlg9k_929a4d2e-02fa-4d5f-ace0-0ad3d41614b7/konnectivity-agent/0.log" limit=1024
Nov 02 01:24:44.609057 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:44.609022    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vlg9k_929a4d2e-02fa-4d5f-ace0-0ad3d41614b7/konnectivity-agent/0.log" limit=1024
Nov 02 01:24:44.628233 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:44.628213    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9lvcx_d352c5c1-44a0-4929-a43a-30c723958ce7/node-ca/0.log"
Nov 02 01:24:44.968170 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:44.968095    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:24:44.968170 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:44.968115    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:44.968375 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:44.968108    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:44.968375 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:44.968192    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4jf8" podUID="4240f2d9-dd99-4ca9-8447-7706cebcfa08"
Nov 02 01:24:44.968375 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:44.968303    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfwwn" podUID="9eb605c8-f995-49ae-a1b8-06aa55160b13"
Nov 02 01:24:44.968375 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:44.968361    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kqsh" podUID="26cf53e3-9cf3-4ad5-afe2-5c241f6749ad"
Nov 02 01:24:45.113781 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:45.113751    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmzfl" event={"ID":"b6238ba5-f475-4e61-94d8-7d7bc13b2458","Type":"ContainerStarted","Data":"28aa63307258b60202bc1676f744a518a3ba0b4184dfec581ca991d58e089cc0"}
Nov 02 01:24:45.251349 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:45.251234    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kmzfl" podStartSLOduration=8.143063076 podStartE2EDuration="21.251218383s" totalImagesPullingTime="13.108155307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:45.250949944 +0000 UTC m=+21.712467619" watchObservedRunningTime="2026-11-02 01:24:45.251218383 +0000 UTC m=+21.712736055"
Nov 02 01:24:46.120205 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:46.119948    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9" event={"ID":"ef8451f5-e54e-42e3-aad7-16e13fb70d81","Type":"ContainerStarted","Data":"800bf979f8c1683abe23dbd9976e55ec51a22294d11e638b8f0e898c9551382a"}
Nov 02 01:24:46.387804 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:46.387677    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9" podStartSLOduration=6.7307569 podStartE2EDuration="22.387664779s" totalImagesPullingTime="15.656907879s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:46.337833581 +0000 UTC m=+22.799351280" watchObservedRunningTime="2026-11-02 01:24:46.387664779 +0000 UTC m=+22.849182454"
Nov 02 01:24:46.968658 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:46.968626    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:46.968917 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:46.968704    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:24:46.968917 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:46.968790    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:46.968917 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:46.968801    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kqsh" podUID="26cf53e3-9cf3-4ad5-afe2-5c241f6749ad"
Nov 02 01:24:46.968917 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:46.968875    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfwwn" podUID="9eb605c8-f995-49ae-a1b8-06aa55160b13"
Nov 02 01:24:46.968917 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:46.968901    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4jf8" podUID="4240f2d9-dd99-4ca9-8447-7706cebcfa08"
Nov 02 01:24:47.122563 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:47.122532    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 01:24:47.122944 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:47.122906    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:47.122997 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:47.122990    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:47.140272 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:47.140242    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:47.143763 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:47.143744    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:48.124443 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:48.124415    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 01:24:48.968070 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:48.968032    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:48.968251 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:48.968089    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:48.968251 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:48.968226    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfwwn" podUID="9eb605c8-f995-49ae-a1b8-06aa55160b13"
Nov 02 01:24:48.968251 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:48.968243    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:24:48.968382 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:48.968348    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kqsh" podUID="26cf53e3-9cf3-4ad5-afe2-5c241f6749ad"
Nov 02 01:24:48.968382 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:48.968352    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4jf8" podUID="4240f2d9-dd99-4ca9-8447-7706cebcfa08"
Nov 02 01:24:49.126695 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:49.126658    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 01:24:50.968391 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:50.968357    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:50.968391 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:50.968380    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:50.968831 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:50.968357    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:24:50.968831 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:50.968468    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kqsh" podUID="26cf53e3-9cf3-4ad5-afe2-5c241f6749ad"
Nov 02 01:24:50.968831 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:50.968523    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4jf8" podUID="4240f2d9-dd99-4ca9-8447-7706cebcfa08"
Nov 02 01:24:50.968831 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:50.968598    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfwwn" podUID="9eb605c8-f995-49ae-a1b8-06aa55160b13"
Nov 02 01:24:52.967955 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:52.967915    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:24:52.968401 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:52.967931    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:52.968401 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:52.968040    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:52.968401 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:52.968039    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4jf8" podUID="4240f2d9-dd99-4ca9-8447-7706cebcfa08"
Nov 02 01:24:52.968401 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:52.968108    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kqsh" podUID="26cf53e3-9cf3-4ad5-afe2-5c241f6749ad"
Nov 02 01:24:52.968401 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:52.968186    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfwwn" podUID="9eb605c8-f995-49ae-a1b8-06aa55160b13"
Nov 02 01:24:53.219138 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:53.219045    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:24:53.219344 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:53.219314    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 01:24:53.229612 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:53.229573    2574 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9" podUID="ef8451f5-e54e-42e3-aad7-16e13fb70d81" containerName="ovnkube-controller" probeResult="failure" output=""
Nov 02 01:24:53.238729 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:53.238694    2574 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9" podUID="ef8451f5-e54e-42e3-aad7-16e13fb70d81" containerName="ovnkube-controller" probeResult="failure" output=""
Nov 02 01:24:54.967689 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:54.967655    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:24:54.967689 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:54.967678    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:54.968127 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:54.967655    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:54.968127 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:54.967772    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4jf8" podUID="4240f2d9-dd99-4ca9-8447-7706cebcfa08"
Nov 02 01:24:54.968127 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:54.967832    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kqsh" podUID="26cf53e3-9cf3-4ad5-afe2-5c241f6749ad"
Nov 02 01:24:54.968127 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:54.967920    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfwwn" podUID="9eb605c8-f995-49ae-a1b8-06aa55160b13"
Nov 02 01:24:56.889510 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:56.889470    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs\") pod \"network-metrics-daemon-rfwwn\" (UID: \"9eb605c8-f995-49ae-a1b8-06aa55160b13\") " pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:56.889890 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:56.889529    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret\") pod \"global-pull-secret-syncer-9kqsh\" (UID: \"26cf53e3-9cf3-4ad5-afe2-5c241f6749ad\") " pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:56.889890 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:56.889624    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:24:56.889890 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:56.889624    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:24:56.889890 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:56.889687    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret podName:26cf53e3-9cf3-4ad5-afe2-5c241f6749ad nodeName:}" failed. No retries permitted until 2026-11-02 01:25:28.889671423 +0000 UTC m=+65.351189091 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret") pod "global-pull-secret-syncer-9kqsh" (UID: "26cf53e3-9cf3-4ad5-afe2-5c241f6749ad") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:24:56.889890 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:56.889706    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs podName:9eb605c8-f995-49ae-a1b8-06aa55160b13 nodeName:}" failed. No retries permitted until 2026-11-02 01:25:28.889697133 +0000 UTC m=+65.351214800 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs") pod "network-metrics-daemon-rfwwn" (UID: "9eb605c8-f995-49ae-a1b8-06aa55160b13") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:24:56.968107 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:56.968073    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:24:56.968271 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:56.968163    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4jf8" podUID="4240f2d9-dd99-4ca9-8447-7706cebcfa08"
Nov 02 01:24:56.968271 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:56.968079    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:56.968271 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:56.968190    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:56.968271 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:56.968238    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kqsh" podUID="26cf53e3-9cf3-4ad5-afe2-5c241f6749ad"
Nov 02 01:24:56.968429 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:56.968334    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfwwn" podUID="9eb605c8-f995-49ae-a1b8-06aa55160b13"
Nov 02 01:24:56.990316 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:56.990259    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9w59\" (UniqueName: \"kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59\") pod \"network-check-target-w4jf8\" (UID: \"4240f2d9-dd99-4ca9-8447-7706cebcfa08\") " pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:24:56.990430 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:56.990413    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:24:56.990468 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:56.990433    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:24:56.990468 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:56.990442    2574 projected.go:196] Error preparing data for projected volume kube-api-access-p9w59 for pod openshift-network-diagnostics/network-check-target-w4jf8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:24:56.990526 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:56.990487    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59 podName:4240f2d9-dd99-4ca9-8447-7706cebcfa08 nodeName:}" failed. No retries permitted until 2026-11-02 01:25:28.990475133 +0000 UTC m=+65.451992787 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-p9w59" (UniqueName: "kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59") pod "network-check-target-w4jf8" (UID: "4240f2d9-dd99-4ca9-8447-7706cebcfa08") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:24:58.968014 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:58.967979    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:24:58.968454 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:58.968021    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:24:58.968454 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:24:58.968107    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:24:58.968454 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:58.968124    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4jf8" podUID="4240f2d9-dd99-4ca9-8447-7706cebcfa08"
Nov 02 01:24:58.968454 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:58.968180    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfwwn" podUID="9eb605c8-f995-49ae-a1b8-06aa55160b13"
Nov 02 01:24:58.968454 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:24:58.968233    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kqsh" podUID="26cf53e3-9cf3-4ad5-afe2-5c241f6749ad"
Nov 02 01:25:00.968424 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:00.968386    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:25:00.968424 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:00.968412    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:25:00.968867 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:00.968386    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:25:00.968867 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:25:00.968544    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfwwn" podUID="9eb605c8-f995-49ae-a1b8-06aa55160b13"
Nov 02 01:25:00.968867 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:25:00.968594    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kqsh" podUID="26cf53e3-9cf3-4ad5-afe2-5c241f6749ad"
Nov 02 01:25:00.968867 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:25:00.968654    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4jf8" podUID="4240f2d9-dd99-4ca9-8447-7706cebcfa08"
Nov 02 01:25:02.967733 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:02.967701    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:25:02.968124 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:02.967711    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:25:02.968124 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:25:02.967807    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kqsh" podUID="26cf53e3-9cf3-4ad5-afe2-5c241f6749ad"
Nov 02 01:25:02.968124 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:25:02.967904    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfwwn" podUID="9eb605c8-f995-49ae-a1b8-06aa55160b13"
Nov 02 01:25:02.968124 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:02.967711    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:25:02.968124 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:25:02.967981    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4jf8" podUID="4240f2d9-dd99-4ca9-8447-7706cebcfa08"
Nov 02 01:25:03.541545 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:03.541513    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9kqsh"]
Nov 02 01:25:03.541853 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:03.541831    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:25:03.541986 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:25:03.541961    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kqsh" podUID="26cf53e3-9cf3-4ad5-afe2-5c241f6749ad"
Nov 02 01:25:03.542424 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:03.542401    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rfwwn"]
Nov 02 01:25:03.542558 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:03.542506    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:25:03.542681 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:25:03.542653    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfwwn" podUID="9eb605c8-f995-49ae-a1b8-06aa55160b13"
Nov 02 01:25:03.543218 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:03.543193    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w4jf8"]
Nov 02 01:25:03.543442 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:03.543423    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:25:03.543542 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:25:03.543522    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4jf8" podUID="4240f2d9-dd99-4ca9-8447-7706cebcfa08"
Nov 02 01:25:04.968085 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:04.967842    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:25:04.968521 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:04.967912    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:25:04.968521 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:25:04.968102    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kqsh" podUID="26cf53e3-9cf3-4ad5-afe2-5c241f6749ad"
Nov 02 01:25:04.968521 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:25:04.968206    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4jf8" podUID="4240f2d9-dd99-4ca9-8447-7706cebcfa08"
Nov 02 01:25:04.968521 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:04.967920    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:25:04.968521 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:25:04.968337    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfwwn" podUID="9eb605c8-f995-49ae-a1b8-06aa55160b13"
Nov 02 01:25:06.968434 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:06.968393    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:25:06.968434 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:06.968410    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:25:06.968952 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:06.968410    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:25:06.968952 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:25:06.968503    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kqsh" podUID="26cf53e3-9cf3-4ad5-afe2-5c241f6749ad"
Nov 02 01:25:06.968952 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:25:06.968574    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4jf8" podUID="4240f2d9-dd99-4ca9-8447-7706cebcfa08"
Nov 02 01:25:06.968952 ip-10-0-136-53 kubenswrapper[2574]: E1102 01:25:06.968639    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfwwn" podUID="9eb605c8-f995-49ae-a1b8-06aa55160b13"
Nov 02 01:25:07.828784 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:07.828760    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeReady"
Nov 02 01:25:07.828906 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:07.828886    2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 01:25:07.987862 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:07.986592    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mjwbd"]
Nov 02 01:25:07.989212 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:07.989195    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bncw7"]
Nov 02 01:25:07.989353 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:07.989335    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mjwbd"
Nov 02 01:25:07.990540 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:07.990522    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bncw7"
Nov 02 01:25:07.993902 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:07.993030    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vrpsb"]
Nov 02 01:25:07.995625 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:07.995606    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mjwbd"]
Nov 02 01:25:07.995758 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:07.995733    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:08.005128 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.005096    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bncw7"]
Nov 02 01:25:08.037661 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.037629    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vrpsb"]
Nov 02 01:25:08.043751 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.043725    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 01:25:08.043923 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.043846    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 01:25:08.043981 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.043952    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 01:25:08.044050 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.044032    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 01:25:08.044102 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.044051    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 01:25:08.044102 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.044071    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q5j6l\""
Nov 02 01:25:08.044102 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.044074    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-tl75j\""
Nov 02 01:25:08.047708 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.047696    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 01:25:08.048491 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.048477    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 01:25:08.053327 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.053307    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l67lg\""
Nov 02 01:25:08.053373 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.053342    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 01:25:08.053373 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.053351    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 01:25:08.067774 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.067745    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa-config-volume\") pod \"dns-default-mjwbd\" (UID: \"dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa\") " pod="openshift-dns/dns-default-mjwbd"
Nov 02 01:25:08.067844 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.067805    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jd7v\" (UniqueName: \"kubernetes.io/projected/23f2f356-b80a-4e8c-a002-c09c006da8a1-kube-api-access-6jd7v\") pod \"ingress-canary-bncw7\" (UID: \"23f2f356-b80a-4e8c-a002-c09c006da8a1\") " pod="openshift-ingress-canary/ingress-canary-bncw7"
Nov 02 01:25:08.067894 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.067878    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa-tmp-dir\") pod \"dns-default-mjwbd\" (UID: \"dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa\") " pod="openshift-dns/dns-default-mjwbd"
Nov 02 01:25:08.067928 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.067915    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23f2f356-b80a-4e8c-a002-c09c006da8a1-cert\") pod \"ingress-canary-bncw7\" (UID: \"23f2f356-b80a-4e8c-a002-c09c006da8a1\") " pod="openshift-ingress-canary/ingress-canary-bncw7"
Nov 02 01:25:08.067966 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.067936    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5cc4f303-fe62-44e7-ac7c-719defde064e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vrpsb\" (UID: \"5cc4f303-fe62-44e7-ac7c-719defde064e\") " pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:08.068001 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.067964    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa-metrics-tls\") pod \"dns-default-mjwbd\" (UID: \"dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa\") " pod="openshift-dns/dns-default-mjwbd"
Nov 02 01:25:08.068074 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.068047    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dck75\" (UniqueName: \"kubernetes.io/projected/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa-kube-api-access-dck75\") pod \"dns-default-mjwbd\" (UID: \"dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa\") " pod="openshift-dns/dns-default-mjwbd"
Nov 02 01:25:08.068157 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.068144    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5cc4f303-fe62-44e7-ac7c-719defde064e-data-volume\") pod \"insights-runtime-extractor-vrpsb\" (UID: \"5cc4f303-fe62-44e7-ac7c-719defde064e\") " pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:08.068200 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.068173    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5cc4f303-fe62-44e7-ac7c-719defde064e-crio-socket\") pod \"insights-runtime-extractor-vrpsb\" (UID: \"5cc4f303-fe62-44e7-ac7c-719defde064e\") " pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:08.068200 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.068189    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5cc4f303-fe62-44e7-ac7c-719defde064e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vrpsb\" (UID: \"5cc4f303-fe62-44e7-ac7c-719defde064e\") " pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:08.068264 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.068204    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkznv\" (UniqueName: \"kubernetes.io/projected/5cc4f303-fe62-44e7-ac7c-719defde064e-kube-api-access-mkznv\") pod \"insights-runtime-extractor-vrpsb\" (UID: \"5cc4f303-fe62-44e7-ac7c-719defde064e\") " pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:08.168534 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.168449    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23f2f356-b80a-4e8c-a002-c09c006da8a1-cert\") pod \"ingress-canary-bncw7\" (UID: \"23f2f356-b80a-4e8c-a002-c09c006da8a1\") " pod="openshift-ingress-canary/ingress-canary-bncw7"
Nov 02 01:25:08.168534 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.168490    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5cc4f303-fe62-44e7-ac7c-719defde064e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vrpsb\" (UID: \"5cc4f303-fe62-44e7-ac7c-719defde064e\") " pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:08.168745 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.168622    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa-metrics-tls\") pod \"dns-default-mjwbd\" (UID: \"dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa\") " pod="openshift-dns/dns-default-mjwbd"
Nov 02 01:25:08.168745 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.168658    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dck75\" (UniqueName: \"kubernetes.io/projected/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa-kube-api-access-dck75\") pod \"dns-default-mjwbd\" (UID: \"dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa\") " pod="openshift-dns/dns-default-mjwbd"
Nov 02 01:25:08.168745 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.168707    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5cc4f303-fe62-44e7-ac7c-719defde064e-data-volume\") pod \"insights-runtime-extractor-vrpsb\" (UID: \"5cc4f303-fe62-44e7-ac7c-719defde064e\") " pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:08.168877 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.168746    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5cc4f303-fe62-44e7-ac7c-719defde064e-crio-socket\") pod \"insights-runtime-extractor-vrpsb\" (UID: \"5cc4f303-fe62-44e7-ac7c-719defde064e\") " pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:08.168877 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.168775    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5cc4f303-fe62-44e7-ac7c-719defde064e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vrpsb\" (UID: \"5cc4f303-fe62-44e7-ac7c-719defde064e\") " pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:08.169109 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.169082    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5cc4f303-fe62-44e7-ac7c-719defde064e-data-volume\") pod \"insights-runtime-extractor-vrpsb\" (UID: \"5cc4f303-fe62-44e7-ac7c-719defde064e\") " pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:08.169182 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.169132    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5cc4f303-fe62-44e7-ac7c-719defde064e-crio-socket\") pod \"insights-runtime-extractor-vrpsb\" (UID: \"5cc4f303-fe62-44e7-ac7c-719defde064e\") " pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:08.169233 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.169203    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkznv\" (UniqueName: \"kubernetes.io/projected/5cc4f303-fe62-44e7-ac7c-719defde064e-kube-api-access-mkznv\") pod \"insights-runtime-extractor-vrpsb\" (UID: \"5cc4f303-fe62-44e7-ac7c-719defde064e\") " pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:08.169303 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.169253    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa-config-volume\") pod \"dns-default-mjwbd\" (UID: \"dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa\") " pod="openshift-dns/dns-default-mjwbd"
Nov 02 01:25:08.169367 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.169350    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jd7v\" (UniqueName: \"kubernetes.io/projected/23f2f356-b80a-4e8c-a002-c09c006da8a1-kube-api-access-6jd7v\") pod \"ingress-canary-bncw7\" (UID: \"23f2f356-b80a-4e8c-a002-c09c006da8a1\") " pod="openshift-ingress-canary/ingress-canary-bncw7"
Nov 02 01:25:08.169424 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.169392    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa-tmp-dir\") pod \"dns-default-mjwbd\" (UID: \"dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa\") " pod="openshift-dns/dns-default-mjwbd"
Nov 02 01:25:08.169479 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.169423    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5cc4f303-fe62-44e7-ac7c-719defde064e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vrpsb\" (UID: \"5cc4f303-fe62-44e7-ac7c-719defde064e\") " pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:08.169667 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.169650    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa-tmp-dir\") pod \"dns-default-mjwbd\" (UID: \"dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa\") " pod="openshift-dns/dns-default-mjwbd"
Nov 02 01:25:08.170087 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.170066    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa-config-volume\") pod \"dns-default-mjwbd\" (UID: \"dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa\") " pod="openshift-dns/dns-default-mjwbd"
Nov 02 01:25:08.173073 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.173048    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa-metrics-tls\") pod \"dns-default-mjwbd\" (UID: \"dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa\") " pod="openshift-dns/dns-default-mjwbd"
Nov 02 01:25:08.173172 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.173100    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23f2f356-b80a-4e8c-a002-c09c006da8a1-cert\") pod \"ingress-canary-bncw7\" (UID: \"23f2f356-b80a-4e8c-a002-c09c006da8a1\") " pod="openshift-ingress-canary/ingress-canary-bncw7"
Nov 02 01:25:08.173324 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.173307    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5cc4f303-fe62-44e7-ac7c-719defde064e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vrpsb\" (UID: \"5cc4f303-fe62-44e7-ac7c-719defde064e\") " pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:08.231821 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.231787    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkznv\" (UniqueName: \"kubernetes.io/projected/5cc4f303-fe62-44e7-ac7c-719defde064e-kube-api-access-mkznv\") pod \"insights-runtime-extractor-vrpsb\" (UID: \"5cc4f303-fe62-44e7-ac7c-719defde064e\") " pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:08.231965 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.231829    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dck75\" (UniqueName: \"kubernetes.io/projected/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa-kube-api-access-dck75\") pod \"dns-default-mjwbd\" (UID: \"dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa\") " pod="openshift-dns/dns-default-mjwbd"
Nov 02 01:25:08.231965 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.231855    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jd7v\" (UniqueName: \"kubernetes.io/projected/23f2f356-b80a-4e8c-a002-c09c006da8a1-kube-api-access-6jd7v\") pod \"ingress-canary-bncw7\" (UID: \"23f2f356-b80a-4e8c-a002-c09c006da8a1\") " pod="openshift-ingress-canary/ingress-canary-bncw7"
Nov 02 01:25:08.307935 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.307899    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mjwbd"
Nov 02 01:25:08.312584 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.312567    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bncw7"
Nov 02 01:25:08.319428 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.319405    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:08.376461 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.376105    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-86d9455949-gsxx6"]
Nov 02 01:25:08.378635 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.378611    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:08.431125 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.431039    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 02 01:25:08.438145 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.438116    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 02 01:25:08.439425 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.439401    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86d9455949-gsxx6"]
Nov 02 01:25:08.442649 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.442629    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lnsht\""
Nov 02 01:25:08.443213 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.443193    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 02 01:25:08.443260 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.443240    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 02 01:25:08.472661 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.472622    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-oauth-config\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:08.472790 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.472694    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-service-ca\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:08.472826 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.472782    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/887b1b53-10f5-4392-89a5-f90a431e60ea-tmp\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:08.472826 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.472812    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-oauth-serving-cert\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:08.472907 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.472829    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfdq8\" (UniqueName: \"kubernetes.io/projected/887b1b53-10f5-4392-89a5-f90a431e60ea-kube-api-access-qfdq8\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:08.472950 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.472935    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-serving-cert\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:08.473005 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.472960    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-console-config\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:08.532496 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.532459    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 02 01:25:08.543334 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.543312    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 02 01:25:08.543456 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.543438    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 02 01:25:08.573763 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.573734    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-oauth-config\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:08.573842 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.573786    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-service-ca\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:08.573842 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.573815    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/887b1b53-10f5-4392-89a5-f90a431e60ea-tmp\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:08.573842 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.573831    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-oauth-serving-cert\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:08.573948 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.573846    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfdq8\" (UniqueName: \"kubernetes.io/projected/887b1b53-10f5-4392-89a5-f90a431e60ea-kube-api-access-qfdq8\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:08.573948 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.573881    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-serving-cert\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:08.573948 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.573904    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-console-config\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:08.574662 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.574374    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/887b1b53-10f5-4392-89a5-f90a431e60ea-tmp\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:08.574858 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.574839    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-oauth-serving-cert\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:08.574927 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.574841    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-console-config\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:08.574927 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.574841    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-service-ca\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:08.576735 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.576703    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-oauth-config\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:08.576845 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.576703    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-serving-cert\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:08.655255 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.655221    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfdq8\" (UniqueName: \"kubernetes.io/projected/887b1b53-10f5-4392-89a5-f90a431e60ea-kube-api-access-qfdq8\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:08.692271 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.692200    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:08.758719 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.758462    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bncw7"]
Nov 02 01:25:08.762267 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:25:08.762241    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f2f356_b80a_4e8c_a002_c09c006da8a1.slice/crio-fa4ccb1c824f7e89a7cd55f1b02889b952b1568c13cd4c3d2435ede62d8e1926 WatchSource:0}: Error finding container fa4ccb1c824f7e89a7cd55f1b02889b952b1568c13cd4c3d2435ede62d8e1926: Status 404 returned error can't find the container with id fa4ccb1c824f7e89a7cd55f1b02889b952b1568c13cd4c3d2435ede62d8e1926
Nov 02 01:25:08.762416 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.762299    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mjwbd"]
Nov 02 01:25:08.764564 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.764541    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vrpsb"]
Nov 02 01:25:08.764954 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:25:08.764913    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd59a74b_47c5_4e19_a2e4_09a8f1ec29aa.slice/crio-61cb902d9136cabebfe4c4727597e4cffe7d9ef29d9beb6b2995f98e1e983c87 WatchSource:0}: Error finding container 61cb902d9136cabebfe4c4727597e4cffe7d9ef29d9beb6b2995f98e1e983c87: Status 404 returned error can't find the container with id 61cb902d9136cabebfe4c4727597e4cffe7d9ef29d9beb6b2995f98e1e983c87
Nov 02 01:25:08.766016 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:25:08.765961    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc4f303_fe62_44e7_ac7c_719defde064e.slice/crio-804900eb52daea113a487364add4a6b182e6bdb792698e117250338b225f0a0c WatchSource:0}: Error finding container 804900eb52daea113a487364add4a6b182e6bdb792698e117250338b225f0a0c: Status 404 returned error can't find the container with id 804900eb52daea113a487364add4a6b182e6bdb792698e117250338b225f0a0c
Nov 02 01:25:08.967965 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.967839    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:25:08.967965 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.967919    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:25:08.967965 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:08.967937    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:25:09.034049 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:09.034017    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 01:25:09.034761 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:09.034053    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 01:25:09.034761 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:09.034195    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 01:25:09.034761 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:09.034197    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qtccq\""
Nov 02 01:25:09.034761 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:09.034257    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 01:25:09.034761 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:09.034369    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-82k56\""
Nov 02 01:25:09.034761 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:09.034482    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6qqtd\""
Nov 02 01:25:09.069010 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:09.068977    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86d9455949-gsxx6"]
Nov 02 01:25:09.073511 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:25:09.073472    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod887b1b53_10f5_4392_89a5_f90a431e60ea.slice/crio-94d9247bfa65f5f7793065e3f27f3ecfd7a081eaca2b2e6276d4acf815e12d9b WatchSource:0}: Error finding container 94d9247bfa65f5f7793065e3f27f3ecfd7a081eaca2b2e6276d4acf815e12d9b: Status 404 returned error can't find the container with id 94d9247bfa65f5f7793065e3f27f3ecfd7a081eaca2b2e6276d4acf815e12d9b
Nov 02 01:25:09.170835 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:09.170803    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vrpsb" event={"ID":"5cc4f303-fe62-44e7-ac7c-719defde064e","Type":"ContainerStarted","Data":"45cc4e4974d493c9232398889cb509c8f68c81cc6ed9a0b88f635f81bfb4703d"}
Nov 02 01:25:09.170983 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:09.170894    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vrpsb" event={"ID":"5cc4f303-fe62-44e7-ac7c-719defde064e","Type":"ContainerStarted","Data":"804900eb52daea113a487364add4a6b182e6bdb792698e117250338b225f0a0c"}
Nov 02 01:25:09.171899 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:09.171879    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bncw7" event={"ID":"23f2f356-b80a-4e8c-a002-c09c006da8a1","Type":"ContainerStarted","Data":"fa4ccb1c824f7e89a7cd55f1b02889b952b1568c13cd4c3d2435ede62d8e1926"}
Nov 02 01:25:09.172963 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:09.172942    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mjwbd" event={"ID":"dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa","Type":"ContainerStarted","Data":"61cb902d9136cabebfe4c4727597e4cffe7d9ef29d9beb6b2995f98e1e983c87"}
Nov 02 01:25:09.174015 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:09.173998    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86d9455949-gsxx6" event={"ID":"887b1b53-10f5-4392-89a5-f90a431e60ea","Type":"ContainerStarted","Data":"94d9247bfa65f5f7793065e3f27f3ecfd7a081eaca2b2e6276d4acf815e12d9b"}
Nov 02 01:25:10.180174 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:10.180130    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vrpsb" event={"ID":"5cc4f303-fe62-44e7-ac7c-719defde064e","Type":"ContainerStarted","Data":"b78097207ee371db03cb631ff00b7a73f43c690e568af8be5ded05d80185244c"}
Nov 02 01:25:12.189811 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:12.189770    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mjwbd" event={"ID":"dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa","Type":"ContainerStarted","Data":"833f30ec25476ce264f71a9cd23a3a16854d4a68b243972964bb455c698849c2"}
Nov 02 01:25:12.190248 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:12.189834    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mjwbd" event={"ID":"dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa","Type":"ContainerStarted","Data":"3408116b4c7dad97640eb132416bdddbf1c01359385236984cf8927929ff29b6"}
Nov 02 01:25:12.190248 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:12.189882    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mjwbd"
Nov 02 01:25:12.307575 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:12.307513    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-mjwbd" podStartSLOduration=2.749511775 podStartE2EDuration="5.307493586s" totalImagesPullingTime="2.557981811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:12.306733206 +0000 UTC m=+48.768250943" watchObservedRunningTime="2026-11-02 01:25:12.307493586 +0000 UTC m=+48.769011262"
Nov 02 01:25:13.194342 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.194268    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vrpsb" event={"ID":"5cc4f303-fe62-44e7-ac7c-719defde064e","Type":"ContainerStarted","Data":"415b11ba8cb2e964289a4135690111c9cf9b1e4be779da8e35962aacf8d890b5"}
Nov 02 01:25:13.195843 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.195819    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bncw7" event={"ID":"23f2f356-b80a-4e8c-a002-c09c006da8a1","Type":"ContainerStarted","Data":"16cd13598d12b83bb867056b023a7536f8e3ce5f744557d4f9779a014c649569"}
Nov 02 01:25:13.197365 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.197333    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86d9455949-gsxx6" event={"ID":"887b1b53-10f5-4392-89a5-f90a431e60ea","Type":"ContainerStarted","Data":"7afacb677ed1ee9f59cc7dd7a4e93676a0a6c45d113fd57a4a483efa568eb74b"}
Nov 02 01:25:13.339020 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.338968    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vrpsb" podStartSLOduration=2.907314907 podStartE2EDuration="6.338955982s" totalImagesPullingTime="3.431641075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:25:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:13.338116026 +0000 UTC m=+49.799633712" watchObservedRunningTime="2026-11-02 01:25:13.338955982 +0000 UTC m=+49.800473657"
Nov 02 01:25:13.426487 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.426448    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-cj277"]
Nov 02 01:25:13.428352 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.428332    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277"
Nov 02 01:25:13.434528 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.434502    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"]
Nov 02 01:25:13.436465 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.436448    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:13.456613 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.456562    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-86d9455949-gsxx6" podStartSLOduration=1.582961067 podStartE2EDuration="5.456549456s" totalImagesPullingTime="3.873588389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:13.456343415 +0000 UTC m=+49.917861091" watchObservedRunningTime="2026-11-02 01:25:13.456549456 +0000 UTC m=+49.918067131"
Nov 02 01:25:13.482537 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.482498    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 01:25:13.482752 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.482734    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 02 01:25:13.483446 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.483429    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-2vxps\""
Nov 02 01:25:13.483446 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.483430    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 01:25:13.483588 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.483498    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 02 01:25:13.483588 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.483566    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 01:25:13.488545 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.488523    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-cj277"]
Nov 02 01:25:13.514789 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.514756    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/de6949bb-b875-4ca6-b0d3-19c07290bfae-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:13.514971 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.514812    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc6acc82-57d4-4906-9c57-67d717e89ba3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-cj277\" (UID: \"bc6acc82-57d4-4906-9c57-67d717e89ba3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277"
Nov 02 01:25:13.514971 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.514845    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2n6b\" (UniqueName: \"kubernetes.io/projected/bc6acc82-57d4-4906-9c57-67d717e89ba3-kube-api-access-c2n6b\") pod \"openshift-state-metrics-7bd7489957-cj277\" (UID: \"bc6acc82-57d4-4906-9c57-67d717e89ba3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277"
Nov 02 01:25:13.514971 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.514876    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/de6949bb-b875-4ca6-b0d3-19c07290bfae-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:13.514971 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.514912    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc6acc82-57d4-4906-9c57-67d717e89ba3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-cj277\" (UID: \"bc6acc82-57d4-4906-9c57-67d717e89ba3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277"
Nov 02 01:25:13.515181 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.515041    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnlcv\" (UniqueName: \"kubernetes.io/projected/de6949bb-b875-4ca6-b0d3-19c07290bfae-kube-api-access-jnlcv\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:13.515181 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.515086    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc6acc82-57d4-4906-9c57-67d717e89ba3-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-cj277\" (UID: \"bc6acc82-57d4-4906-9c57-67d717e89ba3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277"
Nov 02 01:25:13.515181 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.515140    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de6949bb-b875-4ca6-b0d3-19c07290bfae-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:13.515323 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.515178    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de6949bb-b875-4ca6-b0d3-19c07290bfae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:13.515323 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.515208    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/de6949bb-b875-4ca6-b0d3-19c07290bfae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:13.518160 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.518136    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s59gt"]
Nov 02 01:25:13.520164 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.520148    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.530126 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.530105    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"]
Nov 02 01:25:13.571746 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.571609    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bncw7" podStartSLOduration=2.709192021 podStartE2EDuration="6.571592868s" totalImagesPullingTime="3.862400847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:13.571010438 +0000 UTC m=+50.032528133" watchObservedRunningTime="2026-11-02 01:25:13.571592868 +0000 UTC m=+50.033110543"
Nov 02 01:25:13.582846 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.582822    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 02 01:25:13.583006 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.582992    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-tlxc4\""
Nov 02 01:25:13.583051 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.582992    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 02 01:25:13.583123 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.583110    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 02 01:25:13.616473 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.616391    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-textfile\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.616473 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.616450    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-accelerators-collector-config\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.616632 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.616539    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnlcv\" (UniqueName: \"kubernetes.io/projected/de6949bb-b875-4ca6-b0d3-19c07290bfae-kube-api-access-jnlcv\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:13.616632 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.616607    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-wtmp\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.616712 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.616668    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de6949bb-b875-4ca6-b0d3-19c07290bfae-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:13.616712 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.616704    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/de6949bb-b875-4ca6-b0d3-19c07290bfae-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:13.616797 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.616734    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac453c3c-9209-4c20-8fe7-07c6f561b855-root\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.616797 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.616763    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac453c3c-9209-4c20-8fe7-07c6f561b855-sys\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.616918 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.616802    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc6acc82-57d4-4906-9c57-67d717e89ba3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-cj277\" (UID: \"bc6acc82-57d4-4906-9c57-67d717e89ba3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277"
Nov 02 01:25:13.616918 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.616846    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2n6b\" (UniqueName: \"kubernetes.io/projected/bc6acc82-57d4-4906-9c57-67d717e89ba3-kube-api-access-c2n6b\") pod \"openshift-state-metrics-7bd7489957-cj277\" (UID: \"bc6acc82-57d4-4906-9c57-67d717e89ba3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277"
Nov 02 01:25:13.616918 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.616879    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft2nz\" (UniqueName: \"kubernetes.io/projected/ac453c3c-9209-4c20-8fe7-07c6f561b855-kube-api-access-ft2nz\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.616918 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.616901    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/de6949bb-b875-4ca6-b0d3-19c07290bfae-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:13.617098 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.617037    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc6acc82-57d4-4906-9c57-67d717e89ba3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-cj277\" (UID: \"bc6acc82-57d4-4906-9c57-67d717e89ba3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277"
Nov 02 01:25:13.617166 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.617097    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc6acc82-57d4-4906-9c57-67d717e89ba3-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-cj277\" (UID: \"bc6acc82-57d4-4906-9c57-67d717e89ba3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277"
Nov 02 01:25:13.617166 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.617120    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.617166 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.617148    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de6949bb-b875-4ca6-b0d3-19c07290bfae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:13.617358 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.617188    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/de6949bb-b875-4ca6-b0d3-19c07290bfae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:13.617358 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.617213    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-tls\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.617358 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.617242    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac453c3c-9209-4c20-8fe7-07c6f561b855-metrics-client-ca\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.617358 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.617262    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/de6949bb-b875-4ca6-b0d3-19c07290bfae-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:13.617571 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.617547    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de6949bb-b875-4ca6-b0d3-19c07290bfae-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:13.617879 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.617849    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc6acc82-57d4-4906-9c57-67d717e89ba3-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-cj277\" (UID: \"bc6acc82-57d4-4906-9c57-67d717e89ba3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277"
Nov 02 01:25:13.617981 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.617856    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/de6949bb-b875-4ca6-b0d3-19c07290bfae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:13.620497 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.620474    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de6949bb-b875-4ca6-b0d3-19c07290bfae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:13.620575 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.620504    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc6acc82-57d4-4906-9c57-67d717e89ba3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-cj277\" (UID: \"bc6acc82-57d4-4906-9c57-67d717e89ba3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277"
Nov 02 01:25:13.620575 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.620524    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc6acc82-57d4-4906-9c57-67d717e89ba3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-cj277\" (UID: \"bc6acc82-57d4-4906-9c57-67d717e89ba3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277"
Nov 02 01:25:13.620689 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.620673    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/de6949bb-b875-4ca6-b0d3-19c07290bfae-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:13.680858 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.680827    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnlcv\" (UniqueName: \"kubernetes.io/projected/de6949bb-b875-4ca6-b0d3-19c07290bfae-kube-api-access-jnlcv\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:13.681020 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.680893    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2n6b\" (UniqueName: \"kubernetes.io/projected/bc6acc82-57d4-4906-9c57-67d717e89ba3-kube-api-access-c2n6b\") pod \"openshift-state-metrics-7bd7489957-cj277\" (UID: \"bc6acc82-57d4-4906-9c57-67d717e89ba3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277"
Nov 02 01:25:13.683561 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.683538    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4dgbb\""
Nov 02 01:25:13.683665 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.683591    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 01:25:13.683665 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.683611    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 01:25:13.683665 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.683621    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 01:25:13.717920 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.717890    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-tls\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.717920 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.717925    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac453c3c-9209-4c20-8fe7-07c6f561b855-metrics-client-ca\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.718161 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.717951    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-textfile\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.718161 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.717986    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-accelerators-collector-config\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.718161 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.718006    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-wtmp\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.718161 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.718044    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac453c3c-9209-4c20-8fe7-07c6f561b855-root\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.718161 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.718075    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac453c3c-9209-4c20-8fe7-07c6f561b855-sys\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.718161 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.718109    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft2nz\" (UniqueName: \"kubernetes.io/projected/ac453c3c-9209-4c20-8fe7-07c6f561b855-kube-api-access-ft2nz\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.718161 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.718160    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac453c3c-9209-4c20-8fe7-07c6f561b855-sys\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.718528 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.718167    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac453c3c-9209-4c20-8fe7-07c6f561b855-root\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.718528 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.718228    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-wtmp\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.718528 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.718266    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.718528 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.718443    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-textfile\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.718717 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.718696    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-accelerators-collector-config\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.718775 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.718761    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac453c3c-9209-4c20-8fe7-07c6f561b855-metrics-client-ca\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.720574 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.720555    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.720772 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.720756    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-tls\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.740523 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.740492    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277"
Nov 02 01:25:13.747806 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.747787    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:13.779548 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.779516    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft2nz\" (UniqueName: \"kubernetes.io/projected/ac453c3c-9209-4c20-8fe7-07c6f561b855-kube-api-access-ft2nz\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.832204 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:13.832162    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:13.842389 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:25:13.842350    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac453c3c_9209_4c20_8fe7_07c6f561b855.slice/crio-0af5a473261fe5bcb54cf153fa5bcb886bc0949c14e8a899c918475a1186efc6 WatchSource:0}: Error finding container 0af5a473261fe5bcb54cf153fa5bcb886bc0949c14e8a899c918475a1186efc6: Status 404 returned error can't find the container with id 0af5a473261fe5bcb54cf153fa5bcb886bc0949c14e8a899c918475a1186efc6
Nov 02 01:25:14.170215 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:14.170171    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-cj277"]
Nov 02 01:25:14.171416 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:14.171397    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"]
Nov 02 01:25:14.173382 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:25:14.173355    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc6acc82_57d4_4906_9c57_67d717e89ba3.slice/crio-228d085461b0f1841a965aa7a3d32d4ea4a62d7135412a81ba882ffb45fd7a2a WatchSource:0}: Error finding container 228d085461b0f1841a965aa7a3d32d4ea4a62d7135412a81ba882ffb45fd7a2a: Status 404 returned error can't find the container with id 228d085461b0f1841a965aa7a3d32d4ea4a62d7135412a81ba882ffb45fd7a2a
Nov 02 01:25:14.173862 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:25:14.173842    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6949bb_b875_4ca6_b0d3_19c07290bfae.slice/crio-3cbbc555c965562205f86366531a5c8550fb5251b3066467acbfdfe46b3be7fc WatchSource:0}: Error finding container 3cbbc555c965562205f86366531a5c8550fb5251b3066467acbfdfe46b3be7fc: Status 404 returned error can't find the container with id 3cbbc555c965562205f86366531a5c8550fb5251b3066467acbfdfe46b3be7fc
Nov 02 01:25:14.201145 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:14.201119    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277" event={"ID":"bc6acc82-57d4-4906-9c57-67d717e89ba3","Type":"ContainerStarted","Data":"228d085461b0f1841a965aa7a3d32d4ea4a62d7135412a81ba882ffb45fd7a2a"}
Nov 02 01:25:14.202138 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:14.202112    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s59gt" event={"ID":"ac453c3c-9209-4c20-8fe7-07c6f561b855","Type":"ContainerStarted","Data":"0af5a473261fe5bcb54cf153fa5bcb886bc0949c14e8a899c918475a1186efc6"}
Nov 02 01:25:14.203534 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:14.203502    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f" event={"ID":"de6949bb-b875-4ca6-b0d3-19c07290bfae","Type":"ContainerStarted","Data":"3cbbc555c965562205f86366531a5c8550fb5251b3066467acbfdfe46b3be7fc"}
Nov 02 01:25:15.210315 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.210256    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277" event={"ID":"bc6acc82-57d4-4906-9c57-67d717e89ba3","Type":"ContainerStarted","Data":"21ec06f253dc3d7f343ce241f63d7e863380df4471fc72916e62fd0a8de8d1ff"}
Nov 02 01:25:15.210708 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.210344    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277" event={"ID":"bc6acc82-57d4-4906-9c57-67d717e89ba3","Type":"ContainerStarted","Data":"7faa903acf745346b05fa309792a858f48be03caf906e772df4a783720f190d6"}
Nov 02 01:25:15.712874 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.712831    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 01:25:15.716120 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.715645    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.768810 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.768541    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 02 01:25:15.768810 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.768618    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 02 01:25:15.768810 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.768637    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 02 01:25:15.768810 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.768648    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 02 01:25:15.768810 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.768622    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 02 01:25:15.768810 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.768618    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 02 01:25:15.768810 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.768763    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jpr5g\""
Nov 02 01:25:15.837461 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.837429    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.837574 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.837479    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdlxn\" (UniqueName: \"kubernetes.io/projected/b24f6847-62c1-4f87-b320-80327be23ca9-kube-api-access-bdlxn\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.837574 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.837511    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b24f6847-62c1-4f87-b320-80327be23ca9-config-out\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.837651 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.837606    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-web-config\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.837756 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.837738    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.837825 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.837774    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b24f6847-62c1-4f87-b320-80327be23ca9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.837825 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.837811    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.837901 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.837843    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.837901 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.837871    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b24f6847-62c1-4f87-b320-80327be23ca9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.837978 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.837934    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b24f6847-62c1-4f87-b320-80327be23ca9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.837978 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.837970    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b24f6847-62c1-4f87-b320-80327be23ca9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.838051 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.837999    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-config-volume\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.838051 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.838024    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.868229 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.868195    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 02 01:25:15.868644 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.868620    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 02 01:25:15.868755 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.868742    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 02 01:25:15.869737 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.869719    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 01:25:15.938495 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.938445    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.938628 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.938502    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b24f6847-62c1-4f87-b320-80327be23ca9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.938628 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.938546    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b24f6847-62c1-4f87-b320-80327be23ca9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.938628 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.938570    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b24f6847-62c1-4f87-b320-80327be23ca9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.938628 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.938588    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-config-volume\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.938628 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.938602    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.938854 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.938634    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.938854 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.938659    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdlxn\" (UniqueName: \"kubernetes.io/projected/b24f6847-62c1-4f87-b320-80327be23ca9-kube-api-access-bdlxn\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.938854 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.938682    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b24f6847-62c1-4f87-b320-80327be23ca9-config-out\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.938854 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.938704    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-web-config\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.938854 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.938746    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.938854 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.938766    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b24f6847-62c1-4f87-b320-80327be23ca9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.938854 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.938789    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.939135 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.939038    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b24f6847-62c1-4f87-b320-80327be23ca9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.939592 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.939344    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b24f6847-62c1-4f87-b320-80327be23ca9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.940554 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.940525    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b24f6847-62c1-4f87-b320-80327be23ca9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.942539 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.942513    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.942922 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.942898    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-config-volume\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.942996 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.942922    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.943064 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.943048    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b24f6847-62c1-4f87-b320-80327be23ca9-config-out\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.943383 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.943364    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.944302 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.944248    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b24f6847-62c1-4f87-b320-80327be23ca9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.944404 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.944372    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.944828 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.944804    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:15.945213 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:15.945195    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-web-config\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:16.000252 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.000227    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdlxn\" (UniqueName: \"kubernetes.io/projected/b24f6847-62c1-4f87-b320-80327be23ca9-kube-api-access-bdlxn\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:16.033220 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.033191    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-c68bc9867-x88cx"]
Nov 02 01:25:16.034793 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.034774    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:16.035729 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.035711    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.100880 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.090384    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-evgl73d9t320t\""
Nov 02 01:25:16.100880 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.092512    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 02 01:25:16.100880 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.092607    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 02 01:25:16.100880 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.092680    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 02 01:25:16.100880 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.092811    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 02 01:25:16.100880 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.093059    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 02 01:25:16.100880 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.099335    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-txd4k\""
Nov 02 01:25:16.100880 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.099477    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c68bc9867-x88cx"]
Nov 02 01:25:16.140988 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.140954    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-tmp\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.141160 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.140997    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-tls\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.141160 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.141021    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-grpc-tls\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.141250 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.141153    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.141250 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.141189    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-metrics-client-ca\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.141250 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.141209    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.141411 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.141348    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.141411 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.141383    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.141486 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.141414    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp2pv\" (UniqueName: \"kubernetes.io/projected/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-kube-api-access-pp2pv\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.215826 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.215793    2574 generic.go:356] "Generic (PLEG): container finished" podID="ac453c3c-9209-4c20-8fe7-07c6f561b855" containerID="a9bb81938ecc9638a84f5809bcd6a532e79b1a0ca7a1d9164c44b6240a98e934" exitCode=0
Nov 02 01:25:16.216260 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.215869    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s59gt" event={"ID":"ac453c3c-9209-4c20-8fe7-07c6f561b855","Type":"ContainerDied","Data":"a9bb81938ecc9638a84f5809bcd6a532e79b1a0ca7a1d9164c44b6240a98e934"}
Nov 02 01:25:16.218571 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.218525    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f" event={"ID":"de6949bb-b875-4ca6-b0d3-19c07290bfae","Type":"ContainerStarted","Data":"7ab25f08bff22f2535a00c1c5d10fc5b25894e96a1d26f91388adc536dcb7f5e"}
Nov 02 01:25:16.218680 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.218646    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f" event={"ID":"de6949bb-b875-4ca6-b0d3-19c07290bfae","Type":"ContainerStarted","Data":"66e2836994c69498e9933df89e156eb55564c385f5bd5a158e85f25dd4f4c8e6"}
Nov 02 01:25:16.218739 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.218684    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f" event={"ID":"de6949bb-b875-4ca6-b0d3-19c07290bfae","Type":"ContainerStarted","Data":"3fbd0ec65c3583e6ad70f8bd697a86d0488a8009337d344861996ea0d4ff13c0"}
Nov 02 01:25:16.224445 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.224410    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277" event={"ID":"bc6acc82-57d4-4906-9c57-67d717e89ba3","Type":"ContainerStarted","Data":"86e3ce20574e34d8b1bd2f385d64b77f9890987c4e512249ffb03bcc4bc2930d"}
Nov 02 01:25:16.242831 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.242803    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.242971 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.242839    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-metrics-client-ca\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.242971 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.242859    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.242971 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.242880    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.242971 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.242896    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.242971 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.242910    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp2pv\" (UniqueName: \"kubernetes.io/projected/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-kube-api-access-pp2pv\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.242971 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.242953    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-tmp\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.243224 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.242990    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-tls\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.243224 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.243027    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-grpc-tls\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.243860 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.243832    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-tmp\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.243963 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.243889    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-metrics-client-ca\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.245732 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.245680    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.245732 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.245712    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.246313 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.246261    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.246552 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.246526    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-grpc-tls\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.246632 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.246546    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.246849 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.246832    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-tls\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.304014 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.303985    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp2pv\" (UniqueName: \"kubernetes.io/projected/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-kube-api-access-pp2pv\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.351091 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.351056    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:16.451588 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.451534    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277" podStartSLOduration=1.946033976 podStartE2EDuration="3.451518831s" totalImagesPullingTime="1.505484855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:16.450763978 +0000 UTC m=+52.912281656" watchObservedRunningTime="2026-11-02 01:25:16.451518831 +0000 UTC m=+52.913036506"
Nov 02 01:25:16.469570 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.469541    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 01:25:16.471808 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:25:16.471782    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24f6847_62c1_4f87_b320_80327be23ca9.slice/crio-31f315b40def8816fd32f0a2dbb100de956ef67ba6cfea10763fefa9102a5503 WatchSource:0}: Error finding container 31f315b40def8816fd32f0a2dbb100de956ef67ba6cfea10763fefa9102a5503: Status 404 returned error can't find the container with id 31f315b40def8816fd32f0a2dbb100de956ef67ba6cfea10763fefa9102a5503
Nov 02 01:25:16.570183 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.570126    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f" podStartSLOduration=1.869577834 podStartE2EDuration="3.570103497s" totalImagesPullingTime="1.700525663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:16.568394942 +0000 UTC m=+53.029912639" watchObservedRunningTime="2026-11-02 01:25:16.570103497 +0000 UTC m=+53.031621171"
Nov 02 01:25:16.724000 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:16.723977    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c68bc9867-x88cx"]
Nov 02 01:25:16.726625 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:25:16.726598    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf11d2490_7d7e_4ce5_bdd1_8f30ebe09e1d.slice/crio-101a936fe9e4dae209c0b3001269273ffeb0cf18a0a57b71d7be1c16b4b4f09d WatchSource:0}: Error finding container 101a936fe9e4dae209c0b3001269273ffeb0cf18a0a57b71d7be1c16b4b4f09d: Status 404 returned error can't find the container with id 101a936fe9e4dae209c0b3001269273ffeb0cf18a0a57b71d7be1c16b4b4f09d
Nov 02 01:25:17.229924 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:17.229886    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s59gt" event={"ID":"ac453c3c-9209-4c20-8fe7-07c6f561b855","Type":"ContainerStarted","Data":"e9742f1b692babb276f1b5ec22e55301c00df826a57d28481704cfcf26a91e99"}
Nov 02 01:25:17.230516 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:17.229947    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s59gt" event={"ID":"ac453c3c-9209-4c20-8fe7-07c6f561b855","Type":"ContainerStarted","Data":"cad400963969f60990cfbc2069bd846a509089068eb520ac10f29efb158c2fb5"}
Nov 02 01:25:17.231096 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:17.231072    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx" event={"ID":"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d","Type":"ContainerStarted","Data":"101a936fe9e4dae209c0b3001269273ffeb0cf18a0a57b71d7be1c16b4b4f09d"}
Nov 02 01:25:17.234796 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:17.234769    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b24f6847-62c1-4f87-b320-80327be23ca9","Type":"ContainerStarted","Data":"31f315b40def8816fd32f0a2dbb100de956ef67ba6cfea10763fefa9102a5503"}
Nov 02 01:25:17.349153 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:17.349101    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s59gt" podStartSLOduration=2.958984708 podStartE2EDuration="4.349087563s" totalImagesPullingTime="1.29738541s" totalInitContainerRuntime="92.717445ms" isStatefulPod=true podCreationTimestamp="2026-11-02 01:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:17.347767736 +0000 UTC m=+53.809285408" watchObservedRunningTime="2026-11-02 01:25:17.349087563 +0000 UTC m=+53.810605237"
Nov 02 01:25:18.240714 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.240629    2574 generic.go:356] "Generic (PLEG): container finished" podID="b24f6847-62c1-4f87-b320-80327be23ca9" containerID="a2f9b1af7a2b352155f273921041bf5082eb862f39ee2a70bd1946b4ae7d4b77" exitCode=0
Nov 02 01:25:18.241106 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.240720    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b24f6847-62c1-4f87-b320-80327be23ca9","Type":"ContainerDied","Data":"a2f9b1af7a2b352155f273921041bf5082eb862f39ee2a70bd1946b4ae7d4b77"}
Nov 02 01:25:18.378050 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.378017    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"]
Nov 02 01:25:18.381354 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.381333    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:18.435378 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.435341    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 02 01:25:18.435378 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.435367    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 02 01:25:18.435378 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.435356    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 02 01:25:18.435621 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.435419    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 02 01:25:18.435621 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.435442    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fr4ur72tdnr05\""
Nov 02 01:25:18.435621 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.435449    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jpwmt\""
Nov 02 01:25:18.442797 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.442767    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"]
Nov 02 01:25:18.465640 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.465603    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b49b523f-d706-4d6c-aac0-020b2e75c88a-secret-metrics-server-tls\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:18.465792 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.465652    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b49b523f-d706-4d6c-aac0-020b2e75c88a-client-ca-bundle\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:18.465792 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.465749    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b49b523f-d706-4d6c-aac0-020b2e75c88a-audit-log\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:18.465915 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.465805    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b49b523f-d706-4d6c-aac0-020b2e75c88a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:18.465915 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.465887    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzlrg\" (UniqueName: \"kubernetes.io/projected/b49b523f-d706-4d6c-aac0-020b2e75c88a-kube-api-access-jzlrg\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:18.466022 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.465940    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b49b523f-d706-4d6c-aac0-020b2e75c88a-metrics-server-audit-profiles\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:18.466022 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.465971    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b49b523f-d706-4d6c-aac0-020b2e75c88a-secret-metrics-server-client-certs\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:18.567471 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.567381    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b49b523f-d706-4d6c-aac0-020b2e75c88a-secret-metrics-server-tls\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:18.567471 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.567430    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b49b523f-d706-4d6c-aac0-020b2e75c88a-client-ca-bundle\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:18.567471 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.567461    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b49b523f-d706-4d6c-aac0-020b2e75c88a-audit-log\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:18.567834 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.567800    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b49b523f-d706-4d6c-aac0-020b2e75c88a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:18.567962 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.567851    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b49b523f-d706-4d6c-aac0-020b2e75c88a-audit-log\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:18.567962 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.567855    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzlrg\" (UniqueName: \"kubernetes.io/projected/b49b523f-d706-4d6c-aac0-020b2e75c88a-kube-api-access-jzlrg\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:18.567962 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.567906    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b49b523f-d706-4d6c-aac0-020b2e75c88a-metrics-server-audit-profiles\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:18.567962 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.567941    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b49b523f-d706-4d6c-aac0-020b2e75c88a-secret-metrics-server-client-certs\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:18.568472 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.568446    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b49b523f-d706-4d6c-aac0-020b2e75c88a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:18.569265 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.569236    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b49b523f-d706-4d6c-aac0-020b2e75c88a-metrics-server-audit-profiles\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:18.570404 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.570363    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b49b523f-d706-4d6c-aac0-020b2e75c88a-client-ca-bundle\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:18.570515 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.570496    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b49b523f-d706-4d6c-aac0-020b2e75c88a-secret-metrics-server-tls\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:18.570855 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.570835    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b49b523f-d706-4d6c-aac0-020b2e75c88a-secret-metrics-server-client-certs\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:18.629144 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.629102    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzlrg\" (UniqueName: \"kubernetes.io/projected/b49b523f-d706-4d6c-aac0-020b2e75c88a-kube-api-access-jzlrg\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:18.692005 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.691968    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:18.692991 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.692960    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:18.693110 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.693024    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:18.700033 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:18.700011    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:19.128176 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:19.128066    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"]
Nov 02 01:25:19.131172 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:25:19.131148    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb49b523f_d706_4d6c_aac0_020b2e75c88a.slice/crio-8a7840d9f917d98914d9d25551d9543345e78d6cdaa91062c2517ee49ac5782e WatchSource:0}: Error finding container 8a7840d9f917d98914d9d25551d9543345e78d6cdaa91062c2517ee49ac5782e: Status 404 returned error can't find the container with id 8a7840d9f917d98914d9d25551d9543345e78d6cdaa91062c2517ee49ac5782e
Nov 02 01:25:19.244686 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:19.244649    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2" event={"ID":"b49b523f-d706-4d6c-aac0-020b2e75c88a","Type":"ContainerStarted","Data":"8a7840d9f917d98914d9d25551d9543345e78d6cdaa91062c2517ee49ac5782e"}
Nov 02 01:25:19.248692 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:19.248671    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:20.251880 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:20.251806    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx" event={"ID":"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d","Type":"ContainerStarted","Data":"0edeb6bb712cc3e59e18e5914200a722035fc200e2694c275b9eef5da2c05e73"}
Nov 02 01:25:20.251880 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:20.251868    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx" event={"ID":"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d","Type":"ContainerStarted","Data":"fed8be8f96f99dd398c98e4443c7e50e089080a28b98c10a52cb6e4c86809334"}
Nov 02 01:25:20.252257 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:20.251894    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx" event={"ID":"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d","Type":"ContainerStarted","Data":"69d903a1245b0adff095aeae77aad6aa9f3c4b4135aed3413a528f527454bd37"}
Nov 02 01:25:21.259902 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:21.259871    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b24f6847-62c1-4f87-b320-80327be23ca9","Type":"ContainerStarted","Data":"2154e062960bb97fa61add68105bf163c102ca16cb7ff631642808e15955dbb8"}
Nov 02 01:25:21.260337 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:21.259973    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b24f6847-62c1-4f87-b320-80327be23ca9","Type":"ContainerStarted","Data":"e4bd992cf245d9545be341c555b4e9faaf95c09115677dc2d433a11c19f1b0d5"}
Nov 02 01:25:21.260337 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:21.260025    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b24f6847-62c1-4f87-b320-80327be23ca9","Type":"ContainerStarted","Data":"0f4531dc393e10959694446647655bccd77e1e3b837bb5ad6d0764ec41518de3"}
Nov 02 01:25:21.260337 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:21.260055    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b24f6847-62c1-4f87-b320-80327be23ca9","Type":"ContainerStarted","Data":"fef375051d2276b6faa3bffc7ee304a8a508397fc33754e4ef3290a121fe0139"}
Nov 02 01:25:21.260337 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:21.260082    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b24f6847-62c1-4f87-b320-80327be23ca9","Type":"ContainerStarted","Data":"bf7dd96fc431d213b1729dd5e71942568d39bfdedc3c918b18b3bec8ed1de31f"}
Nov 02 01:25:22.199602 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:22.199521    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mjwbd"
Nov 02 01:25:22.266513 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:22.266467    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx" event={"ID":"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d","Type":"ContainerStarted","Data":"b8a999b4b9e57c1f78620e8be9e22ce0dc2ab2cbce3d5c71d6025d93f2495f02"}
Nov 02 01:25:22.266924 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:22.266537    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx" event={"ID":"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d","Type":"ContainerStarted","Data":"5df765fcf10197116d98993f945a8a99b743d8e7d16ea8e06be694857ca56b13"}
Nov 02 01:25:22.266924 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:22.266587    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:22.266924 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:22.266616    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx" event={"ID":"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d","Type":"ContainerStarted","Data":"d7a2aa51a18671266e03f7575dd0b362f67ce627c3224aa3f8eb6551aac6465e"}
Nov 02 01:25:22.270055 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:22.270031    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b24f6847-62c1-4f87-b320-80327be23ca9","Type":"ContainerStarted","Data":"9009b04852505c6f7aecebfd17f4f7b23bc5ee793080eded82aa2c0a36c3da3f"}
Nov 02 01:25:22.271562 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:22.271538    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2" event={"ID":"b49b523f-d706-4d6c-aac0-020b2e75c88a","Type":"ContainerStarted","Data":"132c31959a91e7436954d204299609bdbd2a393d5fb1b31fe3b142deffc161a4"}
Nov 02 01:25:22.460381 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:22.460254    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.635919921 podStartE2EDuration="7.460242358s" totalImagesPullingTime="4.809273445s" totalInitContainerRuntime="15.048992ms" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:22.458553174 +0000 UTC m=+58.920070872" watchObservedRunningTime="2026-11-02 01:25:22.460242358 +0000 UTC m=+58.921760032"
Nov 02 01:25:22.593879 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:22.593823    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx" podStartSLOduration=3.00100381 podStartE2EDuration="7.593808254s" totalImagesPullingTime="4.592804444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:22.592313468 +0000 UTC m=+59.053831168" watchObservedRunningTime="2026-11-02 01:25:22.593808254 +0000 UTC m=+59.055325983"
Nov 02 01:25:22.710778 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:22.710681    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2" podStartSLOduration=2.631111399 podStartE2EDuration="4.710667161s" totalImagesPullingTime="2.079555762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:22.709581017 +0000 UTC m=+59.171098693" watchObservedRunningTime="2026-11-02 01:25:22.710667161 +0000 UTC m=+59.172184835"
Nov 02 01:25:23.240937 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:23.240910    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:28.280854 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:28.280823    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:28.976634 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:28.976596    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret\") pod \"global-pull-secret-syncer-9kqsh\" (UID: \"26cf53e3-9cf3-4ad5-afe2-5c241f6749ad\") " pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:25:28.976822 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:28.976658    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs\") pod \"network-metrics-daemon-rfwwn\" (UID: \"9eb605c8-f995-49ae-a1b8-06aa55160b13\") " pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:25:29.028594 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:29.028559    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 01:25:29.028739 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:29.028617    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 01:25:29.040087 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:29.040048    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs\") pod \"network-metrics-daemon-rfwwn\" (UID: \"9eb605c8-f995-49ae-a1b8-06aa55160b13\") " pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:25:29.040210 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:29.040116    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret\") pod \"global-pull-secret-syncer-9kqsh\" (UID: \"26cf53e3-9cf3-4ad5-afe2-5c241f6749ad\") " pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:25:29.077463 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:29.077420    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9w59\" (UniqueName: \"kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59\") pod \"network-check-target-w4jf8\" (UID: \"4240f2d9-dd99-4ca9-8447-7706cebcfa08\") " pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:25:29.129102 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:29.129067    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 01:25:29.138669 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:29.138635    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6qqtd\""
Nov 02 01:25:29.144037 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:29.144020    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-82k56\""
Nov 02 01:25:29.148147 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:29.148128    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:25:29.153418 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:29.153403    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:25:29.189914 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:29.189882    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 01:25:29.203895 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:29.203831    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9w59\" (UniqueName: \"kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59\") pod \"network-check-target-w4jf8\" (UID: \"4240f2d9-dd99-4ca9-8447-7706cebcfa08\") " pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:25:29.430408 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:29.430377    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qtccq\""
Nov 02 01:25:29.439434 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:29.439412    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:25:29.529035 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:29.529011    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9kqsh"]
Nov 02 01:25:29.531268 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:25:29.531226    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26cf53e3_9cf3_4ad5_afe2_5c241f6749ad.slice/crio-727a3c9aae1952df44a8add24e47b3d47830be1c1ab99729d9895854610cb098 WatchSource:0}: Error finding container 727a3c9aae1952df44a8add24e47b3d47830be1c1ab99729d9895854610cb098: Status 404 returned error can't find the container with id 727a3c9aae1952df44a8add24e47b3d47830be1c1ab99729d9895854610cb098
Nov 02 01:25:29.590542 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:29.590436    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rfwwn"]
Nov 02 01:25:29.593062 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:25:29.593030    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb605c8_f995_49ae_a1b8_06aa55160b13.slice/crio-44577b0e87c6cdea0d415ba1edf6bb51264f55ef8c87add715e7acfb77c1361c WatchSource:0}: Error finding container 44577b0e87c6cdea0d415ba1edf6bb51264f55ef8c87add715e7acfb77c1361c: Status 404 returned error can't find the container with id 44577b0e87c6cdea0d415ba1edf6bb51264f55ef8c87add715e7acfb77c1361c
Nov 02 01:25:29.809164 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:29.809072    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w4jf8"]
Nov 02 01:25:29.811462 ip-10-0-136-53 kubenswrapper[2574]: W1102 01:25:29.811431    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4240f2d9_dd99_4ca9_8447_7706cebcfa08.slice/crio-af9eb3b9954bc7ea61ab6821c900840c6b89c8e6a14a0d27ace14b9eaec5138c WatchSource:0}: Error finding container af9eb3b9954bc7ea61ab6821c900840c6b89c8e6a14a0d27ace14b9eaec5138c: Status 404 returned error can't find the container with id af9eb3b9954bc7ea61ab6821c900840c6b89c8e6a14a0d27ace14b9eaec5138c
Nov 02 01:25:30.300856 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:30.300776    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w4jf8" event={"ID":"4240f2d9-dd99-4ca9-8447-7706cebcfa08","Type":"ContainerStarted","Data":"af9eb3b9954bc7ea61ab6821c900840c6b89c8e6a14a0d27ace14b9eaec5138c"}
Nov 02 01:25:30.302102 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:30.302080    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rfwwn" event={"ID":"9eb605c8-f995-49ae-a1b8-06aa55160b13","Type":"ContainerStarted","Data":"44577b0e87c6cdea0d415ba1edf6bb51264f55ef8c87add715e7acfb77c1361c"}
Nov 02 01:25:30.303188 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:30.303167    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9kqsh" event={"ID":"26cf53e3-9cf3-4ad5-afe2-5c241f6749ad","Type":"ContainerStarted","Data":"727a3c9aae1952df44a8add24e47b3d47830be1c1ab99729d9895854610cb098"}
Nov 02 01:25:31.594480 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:31.594449    2574 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86d9455949-gsxx6"]
Nov 02 01:25:32.317410 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:32.317369    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rfwwn" event={"ID":"9eb605c8-f995-49ae-a1b8-06aa55160b13","Type":"ContainerStarted","Data":"1f0c4a3c969bbc94529282cf459d58e0daac921432c03bd6052381f0d25d4c74"}
Nov 02 01:25:32.317569 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:32.317440    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rfwwn" event={"ID":"9eb605c8-f995-49ae-a1b8-06aa55160b13","Type":"ContainerStarted","Data":"438e366ca9733801e329d6b83bb4edd2561ab126f964d732c5e5cf8d51244881"}
Nov 02 01:25:32.434965 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:32.434904    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rfwwn" podStartSLOduration=66.582817057 podStartE2EDuration="1m8.434883898s" totalImagesPullingTime="1.852066841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:24:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:32.433117807 +0000 UTC m=+68.894635484" watchObservedRunningTime="2026-11-02 01:25:32.434883898 +0000 UTC m=+68.896401575"
Nov 02 01:25:34.327785 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:34.327749    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w4jf8" event={"ID":"4240f2d9-dd99-4ca9-8447-7706cebcfa08","Type":"ContainerStarted","Data":"7603d49f84399dac662b23ea07f7101fb7e1b150ddb0af2d489b861b4035f97f"}
Nov 02 01:25:34.328310 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:34.327982    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:25:34.463394 ip-10-0-136-53 kubenswrapper[2574]: I1102 01:25:34.463332    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-w4jf8" podStartSLOduration=66.834992817 podStartE2EDuration="1m10.463311189s" totalImagesPullingTime="3.628318372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:24:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:34.461726466 +0000 UTC m=+70.923244134" watchObservedRunningTime="2026-11-02 01:25:34.463311189 +0000 UTC m=+70.924828866"
Nov 02 01:25:35.039984 ip-10-0-136-53 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 01:25:35.049730 ip-10-0-136-53 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 01:25:35.049902 ip-10-0-136-53 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 01:25:35.050080 ip-10-0-136-53 systemd[1]: kubelet.service: Consumed 4.449s CPU time.
Nov 02 01:25:35.061956 ip-10-0-136-53 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 01:25:35.126192 ip-10-0-136-53 kubenswrapper[5639]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:25:35.126192 ip-10-0-136-53 kubenswrapper[5639]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 01:25:35.126192 ip-10-0-136-53 kubenswrapper[5639]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:25:35.126192 ip-10-0-136-53 kubenswrapper[5639]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:25:35.130145 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130045    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:25:35.130145 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130143    5639 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:25:35.130145 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130147    5639 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:25:35.130145 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130150    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:25:35.130145 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130154    5639 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:25:35.130145 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130156    5639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:25:35.130431 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130159    5639 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:25:35.130431 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130162    5639 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:25:35.130431 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130165    5639 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:25:35.130431 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130167    5639 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:25:35.130431 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130170    5639 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:25:35.130431 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130172    5639 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:25:35.130431 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130175    5639 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:25:35.130431 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130177    5639 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:25:35.130431 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130180    5639 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:25:35.130431 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130185    5639 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:25:35.130431 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130189    5639 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:25:35.130431 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130192    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:25:35.130431 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130195    5639 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:25:35.130431 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130198    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:25:35.130431 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130201    5639 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:25:35.130431 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130203    5639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:25:35.130431 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130205    5639 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:25:35.130431 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130208    5639 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:25:35.130431 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130210    5639 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:25:35.130431 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130214    5639 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:25:35.130928 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130217    5639 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:25:35.130928 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130219    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:25:35.130928 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130222    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:25:35.130928 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130225    5639 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:25:35.130928 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130227    5639 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:25:35.130928 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130230    5639 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:25:35.130928 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130234    5639 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:25:35.130928 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130236    5639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:25:35.130928 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130239    5639 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:25:35.130928 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130241    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:25:35.130928 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130244    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:25:35.130928 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130246    5639 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:25:35.130928 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130249    5639 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:25:35.130928 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130251    5639 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:25:35.130928 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130254    5639 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:25:35.130928 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130257    5639 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:25:35.130928 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130260    5639 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:25:35.130928 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130263    5639 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:25:35.130928 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130266    5639 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:25:35.131451 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130268    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:25:35.131451 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130271    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:25:35.131451 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130274    5639 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:25:35.131451 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130299    5639 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:25:35.131451 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130303    5639 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:25:35.131451 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130305    5639 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:25:35.131451 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130308    5639 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:25:35.131451 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130310    5639 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:25:35.131451 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130313    5639 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:25:35.131451 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130323    5639 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:25:35.131451 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130327    5639 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:25:35.131451 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130330    5639 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:25:35.131451 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130333    5639 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:25:35.131451 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130336    5639 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:25:35.131451 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130339    5639 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:25:35.131451 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130341    5639 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:25:35.131451 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130344    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:25:35.131451 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130346    5639 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:25:35.131451 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130350    5639 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:25:35.131451 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130353    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:25:35.131952 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130356    5639 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:25:35.131952 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130359    5639 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:25:35.131952 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130361    5639 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:25:35.131952 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130364    5639 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:25:35.131952 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130367    5639 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:25:35.131952 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130369    5639 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:25:35.131952 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130371    5639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:25:35.131952 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130374    5639 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:25:35.131952 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130376    5639 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:25:35.131952 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130380    5639 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:25:35.131952 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130383    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:25:35.131952 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130385    5639 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:25:35.131952 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130388    5639 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:25:35.131952 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130390    5639 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:25:35.131952 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130393    5639 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:25:35.131952 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130395    5639 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:25:35.131952 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130398    5639 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:25:35.131952 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130401    5639 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:25:35.131952 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130403    5639 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:25:35.131952 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130406    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:25:35.132509 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130408    5639 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:25:35.132509 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130411    5639 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:25:35.132509 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130414    5639 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:25:35.132509 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130416    5639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:25:35.132509 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130419    5639 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:25:35.132509 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130422    5639 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:25:35.132509 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130424    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:25:35.132509 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130427    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:25:35.132509 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130431    5639 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:25:35.132509 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130435    5639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:25:35.132509 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130438    5639 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:25:35.132509 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130442    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:25:35.132509 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130446    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:25:35.132509 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130450    5639 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:25:35.132509 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130452    5639 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:25:35.132509 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130454    5639 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:25:35.132509 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130457    5639 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:25:35.132509 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130460    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:25:35.132509 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130462    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:25:35.132967 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130465    5639 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:25:35.132967 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.130468    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:25:35.132967 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131264    5639 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:25:35.132967 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131272    5639 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:25:35.132967 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131296    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:25:35.132967 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131301    5639 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:25:35.132967 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131306    5639 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:25:35.132967 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131310    5639 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:25:35.132967 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131313    5639 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:25:35.132967 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131317    5639 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:25:35.132967 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131321    5639 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:25:35.132967 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131325    5639 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:25:35.132967 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131329    5639 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:25:35.132967 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131333    5639 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:25:35.132967 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131336    5639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:25:35.132967 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131339    5639 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:25:35.132967 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131342    5639 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:25:35.132967 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131345    5639 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:25:35.132967 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131347    5639 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:25:35.133455 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131350    5639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:25:35.133455 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131353    5639 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:25:35.133455 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131356    5639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:25:35.133455 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131359    5639 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:25:35.133455 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131361    5639 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:25:35.133455 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131364    5639 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:25:35.133455 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131367    5639 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:25:35.133455 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131369    5639 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:25:35.133455 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131372    5639 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:25:35.133455 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131374    5639 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:25:35.133455 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131377    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:25:35.133455 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131380    5639 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:25:35.133455 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131382    5639 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:25:35.133455 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131385    5639 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:25:35.133455 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131387    5639 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:25:35.133455 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131389    5639 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:25:35.133455 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131393    5639 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:25:35.133455 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131398    5639 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:25:35.133455 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131401    5639 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:25:35.133955 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131404    5639 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:25:35.133955 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131407    5639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:25:35.133955 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131409    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:25:35.133955 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131412    5639 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:25:35.133955 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131414    5639 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:25:35.133955 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131418    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:25:35.133955 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131421    5639 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:25:35.133955 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131424    5639 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:25:35.133955 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131427    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:25:35.133955 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131429    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:25:35.133955 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131432    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:25:35.133955 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131435    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:25:35.133955 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131438    5639 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:25:35.133955 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131440    5639 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:25:35.133955 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131443    5639 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:25:35.133955 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131445    5639 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:25:35.133955 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131448    5639 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:25:35.133955 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131450    5639 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:25:35.133955 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131454    5639 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:25:35.133955 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131456    5639 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:25:35.134488 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131459    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:25:35.134488 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131461    5639 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:25:35.134488 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131464    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:25:35.134488 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131467    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:25:35.134488 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131469    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:25:35.134488 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131472    5639 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:25:35.134488 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131474    5639 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:25:35.134488 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131477    5639 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:25:35.134488 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131479    5639 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:25:35.134488 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131482    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:25:35.134488 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131484    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:25:35.134488 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131488    5639 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:25:35.134488 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131491    5639 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:25:35.134488 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131493    5639 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:25:35.134488 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131496    5639 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:25:35.134488 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131498    5639 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:25:35.134488 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131501    5639 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:25:35.134488 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131504    5639 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:25:35.134488 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131506    5639 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:25:35.134488 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131509    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:25:35.134985 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131512    5639 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:25:35.134985 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131515    5639 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:25:35.134985 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131518    5639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:25:35.134985 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131521    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:25:35.134985 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131524    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:25:35.134985 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131526    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:25:35.134985 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131529    5639 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:25:35.134985 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131532    5639 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:25:35.134985 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131534    5639 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:25:35.134985 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131537    5639 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:25:35.134985 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131539    5639 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:25:35.134985 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131542    5639 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:25:35.134985 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131544    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:25:35.134985 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131547    5639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:25:35.134985 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131549    5639 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:25:35.134985 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131552    5639 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:25:35.134985 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131554    5639 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:25:35.134985 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131557    5639 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:25:35.134985 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131559    5639 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:25:35.135483 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131563    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:25:35.135483 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131565    5639 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:25:35.135483 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131568    5639 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:25:35.135483 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131571    5639 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:25:35.135483 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131574    5639 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:25:35.135483 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131577    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:25:35.135483 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131581    5639 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:25:35.135483 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131587    5639 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:25:35.135483 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131591    5639 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:25:35.135483 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131593    5639 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:25:35.135483 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.131597    5639 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:25:35.135483 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132104    5639 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:25:35.135483 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132109    5639 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:25:35.135483 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132113    5639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:25:35.135483 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132115    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:25:35.135483 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132118    5639 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:25:35.135483 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132121    5639 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:25:35.135483 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132123    5639 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:25:35.135483 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132126    5639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:25:35.135947 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132128    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:25:35.135947 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132131    5639 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:25:35.135947 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132133    5639 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:25:35.135947 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132136    5639 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:25:35.135947 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132138    5639 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:25:35.135947 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132141    5639 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:25:35.135947 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132143    5639 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:25:35.135947 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132146    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:25:35.135947 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132148    5639 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:25:35.135947 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132151    5639 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:25:35.135947 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132154    5639 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:25:35.135947 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132156    5639 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:25:35.135947 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132159    5639 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:25:35.135947 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132161    5639 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:25:35.135947 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132164    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:25:35.135947 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132166    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:25:35.135947 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132169    5639 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:25:35.135947 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132172    5639 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:25:35.135947 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132175    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:25:35.136458 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132177    5639 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:25:35.136458 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132180    5639 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:25:35.136458 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132182    5639 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:25:35.136458 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132184    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:25:35.136458 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132187    5639 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:25:35.136458 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132190    5639 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:25:35.136458 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132193    5639 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:25:35.136458 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132196    5639 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:25:35.136458 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132199    5639 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:25:35.136458 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132201    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:25:35.136458 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132204    5639 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:25:35.136458 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132207    5639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:25:35.136458 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132209    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:25:35.136458 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132212    5639 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:25:35.136458 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132214    5639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:25:35.136458 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132217    5639 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:25:35.136458 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132219    5639 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:25:35.136458 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132222    5639 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:25:35.136458 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132224    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:25:35.136458 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132227    5639 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:25:35.136951 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132229    5639 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:25:35.136951 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132232    5639 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:25:35.136951 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132234    5639 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:25:35.136951 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132238    5639 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:25:35.136951 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132241    5639 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:25:35.136951 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132244    5639 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:25:35.136951 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132247    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:25:35.136951 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132249    5639 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:25:35.136951 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132252    5639 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:25:35.136951 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132254    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:25:35.136951 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132257    5639 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:25:35.136951 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132260    5639 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:25:35.136951 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132263    5639 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:25:35.136951 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132265    5639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:25:35.136951 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132268    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:25:35.136951 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132271    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:25:35.136951 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132273    5639 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:25:35.136951 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132295    5639 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:25:35.136951 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132298    5639 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:25:35.137450 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132301    5639 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:25:35.137450 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132303    5639 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:25:35.137450 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132309    5639 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:25:35.137450 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132312    5639 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:25:35.137450 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132315    5639 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:25:35.137450 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132318    5639 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:25:35.137450 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132321    5639 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:25:35.137450 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132323    5639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:25:35.137450 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132326    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:25:35.137450 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132328    5639 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:25:35.137450 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132331    5639 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:25:35.137450 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132333    5639 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:25:35.137450 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132336    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:25:35.137450 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132338    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:25:35.137450 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132341    5639 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:25:35.137450 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132343    5639 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:25:35.137450 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132346    5639 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:25:35.137450 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132349    5639 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:25:35.137450 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132351    5639 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:25:35.137925 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132354    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:25:35.137925 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132356    5639 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:25:35.137925 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132359    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:25:35.137925 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132361    5639 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:25:35.137925 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132368    5639 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:25:35.137925 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132370    5639 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:25:35.137925 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132373    5639 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:25:35.137925 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132375    5639 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:25:35.137925 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132378    5639 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:25:35.137925 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132381    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:25:35.137925 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132383    5639 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:25:35.137925 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132386    5639 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:25:35.137925 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132388    5639 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:25:35.137925 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132390    5639 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:25:35.137925 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132393    5639 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:25:35.137925 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132395    5639 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:25:35.137925 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132398    5639 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:25:35.137925 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132401    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:25:35.137925 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132404    5639 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:25:35.137925 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132406    5639 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:25:35.138455 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.132409    5639 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:25:35.138455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132520    5639 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 01:25:35.138455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132542    5639 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 01:25:35.138455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132551    5639 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 01:25:35.138455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132556    5639 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 01:25:35.138455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132561    5639 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 01:25:35.138455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132569    5639 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 01:25:35.138455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132578    5639 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 01:25:35.138455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132582    5639 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 01:25:35.138455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132585    5639 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 01:25:35.138455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132588    5639 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 01:25:35.138455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132591    5639 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 01:25:35.138455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132595    5639 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 01:25:35.138455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132598    5639 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 01:25:35.138455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132601    5639 flags.go:64] FLAG: --cgroup-root=""
Nov 02 01:25:35.138455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132603    5639 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 01:25:35.138455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132608    5639 flags.go:64] FLAG: --client-ca-file=""
Nov 02 01:25:35.138455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132611    5639 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 01:25:35.138455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132614    5639 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 01:25:35.138455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132619    5639 flags.go:64] FLAG: --cluster-domain=""
Nov 02 01:25:35.138455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132621    5639 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 01:25:35.138455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132625    5639 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 01:25:35.138455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132628    5639 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 01:25:35.138455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132631    5639 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 01:25:35.139033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132635    5639 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 01:25:35.139033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132638    5639 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 01:25:35.139033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132641    5639 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 01:25:35.139033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132645    5639 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 01:25:35.139033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132648    5639 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 01:25:35.139033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132651    5639 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 01:25:35.139033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132654    5639 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 01:25:35.139033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132658    5639 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 01:25:35.139033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132660    5639 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 01:25:35.139033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132665    5639 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 01:25:35.139033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132668    5639 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 01:25:35.139033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132671    5639 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 01:25:35.139033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132674    5639 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 01:25:35.139033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132677    5639 flags.go:64] FLAG: --enable-server="true"
Nov 02 01:25:35.139033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132682    5639 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 01:25:35.139033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132687    5639 flags.go:64] FLAG: --event-burst="100"
Nov 02 01:25:35.139033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132690    5639 flags.go:64] FLAG: --event-qps="50"
Nov 02 01:25:35.139033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132693    5639 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 01:25:35.139033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132696    5639 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 01:25:35.139033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132699    5639 flags.go:64] FLAG: --eviction-hard=""
Nov 02 01:25:35.139033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132703    5639 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 01:25:35.139033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132705    5639 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 01:25:35.139033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132709    5639 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 01:25:35.139033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132712    5639 flags.go:64] FLAG: --eviction-soft=""
Nov 02 01:25:35.139033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132716    5639 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 01:25:35.139691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132719    5639 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 01:25:35.139691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132722    5639 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 01:25:35.139691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132725    5639 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 01:25:35.139691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132728    5639 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 01:25:35.139691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132730    5639 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 01:25:35.139691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132733    5639 flags.go:64] FLAG: --feature-gates=""
Nov 02 01:25:35.139691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132737    5639 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 01:25:35.139691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132740    5639 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 01:25:35.139691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132744    5639 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 01:25:35.139691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132747    5639 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 01:25:35.139691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132750    5639 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 01:25:35.139691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132753    5639 flags.go:64] FLAG: --help="false"
Nov 02 01:25:35.139691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132756    5639 flags.go:64] FLAG: --hostname-override="ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:25:35.139691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132759    5639 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 01:25:35.139691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132763    5639 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 01:25:35.139691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132765    5639 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 01:25:35.139691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132769    5639 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 01:25:35.139691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132772    5639 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 01:25:35.139691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132775    5639 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 01:25:35.139691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132778    5639 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 01:25:35.139691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132780    5639 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 01:25:35.139691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132785    5639 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 01:25:35.139691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132787    5639 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 01:25:35.140269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132791    5639 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 01:25:35.140269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132793    5639 flags.go:64] FLAG: --kube-reserved=""
Nov 02 01:25:35.140269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132796    5639 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 01:25:35.140269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132799    5639 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 01:25:35.140269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132802    5639 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 01:25:35.140269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132805    5639 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 01:25:35.140269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132807    5639 flags.go:64] FLAG: --lock-file=""
Nov 02 01:25:35.140269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132810    5639 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 01:25:35.140269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132815    5639 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 01:25:35.140269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132818    5639 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 01:25:35.140269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132824    5639 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 01:25:35.140269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132828    5639 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 01:25:35.140269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132831    5639 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 01:25:35.140269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132833    5639 flags.go:64] FLAG: --logging-format="text"
Nov 02 01:25:35.140269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132837    5639 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 01:25:35.140269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132840    5639 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 01:25:35.140269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132843    5639 flags.go:64] FLAG: --manifest-url=""
Nov 02 01:25:35.140269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132846    5639 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 01:25:35.140269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132850    5639 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 01:25:35.140269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132853    5639 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 01:25:35.140269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132857    5639 flags.go:64] FLAG: --max-pods="110"
Nov 02 01:25:35.140269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132860    5639 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 01:25:35.140269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132863    5639 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 01:25:35.140269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132866    5639 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 01:25:35.140269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132869    5639 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 01:25:35.141042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132872    5639 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 01:25:35.141042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132875    5639 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 01:25:35.141042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132878    5639 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 01:25:35.141042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132885    5639 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 01:25:35.141042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132888    5639 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 01:25:35.141042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132892    5639 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 01:25:35.141042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132895    5639 flags.go:64] FLAG: --pod-cidr=""
Nov 02 01:25:35.141042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132898    5639 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 01:25:35.141042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132901    5639 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 01:25:35.141042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132904    5639 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 01:25:35.141042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132907    5639 flags.go:64] FLAG: --port="10250"
Nov 02 01:25:35.141042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132910    5639 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 01:25:35.141042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132913    5639 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0aae5c789d6075945"
Nov 02 01:25:35.141042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132916    5639 flags.go:64] FLAG: --qos-reserved=""
Nov 02 01:25:35.141042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132919    5639 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 01:25:35.141042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132923    5639 flags.go:64] FLAG: --register-node="true"
Nov 02 01:25:35.141042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132926    5639 flags.go:64] FLAG: --register-with-taints=""
Nov 02 01:25:35.141042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132930    5639 flags.go:64] FLAG: --registry-burst="10"
Nov 02 01:25:35.141042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132933    5639 flags.go:64] FLAG: --registry-qps="5"
Nov 02 01:25:35.141042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132936    5639 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 01:25:35.141042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132939    5639 flags.go:64] FLAG: --reserved-memory=""
Nov 02 01:25:35.141042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132943    5639 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 01:25:35.141042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132946    5639 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 01:25:35.141042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132949    5639 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 01:25:35.141042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132952    5639 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 01:25:35.141042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132955    5639 flags.go:64] FLAG: --runonce="false"
Nov 02 01:25:35.141777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132958    5639 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 01:25:35.141777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132961    5639 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 01:25:35.141777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132964    5639 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 01:25:35.141777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132967    5639 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 01:25:35.141777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132970    5639 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 01:25:35.141777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132974    5639 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 01:25:35.141777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132977    5639 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 01:25:35.141777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132980    5639 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 01:25:35.141777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132983    5639 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 01:25:35.141777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132986    5639 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 01:25:35.141777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132989    5639 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 01:25:35.141777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132993    5639 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 01:25:35.141777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132996    5639 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 01:25:35.141777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.132999    5639 flags.go:64] FLAG: --system-cgroups=""
Nov 02 01:25:35.141777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.133001    5639 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 01:25:35.141777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.133006    5639 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 01:25:35.141777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.133009    5639 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 01:25:35.141777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.133012    5639 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 01:25:35.141777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.133017    5639 flags.go:64] FLAG: --tls-min-version=""
Nov 02 01:25:35.141777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.133020    5639 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 01:25:35.141777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.133022    5639 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 01:25:35.141777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.133026    5639 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 01:25:35.141777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.133029    5639 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 01:25:35.141777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.133032    5639 flags.go:64] FLAG: --v="2"
Nov 02 01:25:35.141777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.133036    5639 flags.go:64] FLAG: --version="false"
Nov 02 01:25:35.142615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.133040    5639 flags.go:64] FLAG: --vmodule=""
Nov 02 01:25:35.142615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.133044    5639 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 01:25:35.142615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.133047    5639 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 01:25:35.142615 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133181    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:25:35.142615 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133184    5639 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:25:35.142615 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133187    5639 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:25:35.142615 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133190    5639 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:25:35.142615 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133192    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:25:35.142615 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133195    5639 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:25:35.142615 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133198    5639 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:25:35.142615 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133200    5639 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:25:35.142615 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133203    5639 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:25:35.142615 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133206    5639 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:25:35.142615 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133209    5639 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:25:35.142615 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133212    5639 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:25:35.142615 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133215    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:25:35.142615 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133217    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:25:35.142615 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133220    5639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:25:35.142615 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133223    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:25:35.142615 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133226    5639 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:25:35.143230 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133229    5639 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:25:35.143230 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133231    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:25:35.143230 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133234    5639 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:25:35.143230 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133236    5639 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:25:35.143230 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133239    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:25:35.143230 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133242    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:25:35.143230 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133244    5639 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:25:35.143230 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133247    5639 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:25:35.143230 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133249    5639 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:25:35.143230 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133252    5639 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:25:35.143230 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133254    5639 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:25:35.143230 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133257    5639 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:25:35.143230 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133260    5639 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:25:35.143230 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133262    5639 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:25:35.143230 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133264    5639 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:25:35.143230 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133267    5639 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:25:35.143230 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133269    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:25:35.143230 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133272    5639 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:25:35.143230 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133291    5639 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:25:35.143230 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133295    5639 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:25:35.143958 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133298    5639 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:25:35.143958 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133302    5639 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:25:35.143958 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133305    5639 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:25:35.143958 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133308    5639 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:25:35.143958 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133311    5639 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:25:35.143958 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133313    5639 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:25:35.143958 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133318    5639 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:25:35.143958 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133321    5639 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:25:35.143958 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133324    5639 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:25:35.143958 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133326    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:25:35.143958 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133330    5639 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:25:35.143958 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133333    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:25:35.143958 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133335    5639 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:25:35.143958 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133338    5639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:25:35.143958 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133340    5639 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:25:35.143958 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133343    5639 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:25:35.143958 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133345    5639 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:25:35.143958 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133348    5639 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:25:35.143958 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133350    5639 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:25:35.144517 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133353    5639 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:25:35.144517 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133355    5639 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:25:35.144517 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133358    5639 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:25:35.144517 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133361    5639 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:25:35.144517 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133363    5639 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:25:35.144517 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133366    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:25:35.144517 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133368    5639 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:25:35.144517 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133371    5639 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:25:35.144517 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133373    5639 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:25:35.144517 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133376    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:25:35.144517 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133378    5639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:25:35.144517 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133381    5639 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:25:35.144517 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133383    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:25:35.144517 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133386    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:25:35.144517 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133389    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:25:35.144517 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133391    5639 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:25:35.144517 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133393    5639 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:25:35.144517 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133396    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:25:35.144517 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133398    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:25:35.144980 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133401    5639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:25:35.144980 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133404    5639 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:25:35.144980 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133406    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:25:35.144980 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133409    5639 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:25:35.144980 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133413    5639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:25:35.144980 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133416    5639 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:25:35.144980 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133418    5639 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:25:35.144980 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133421    5639 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:25:35.144980 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133423    5639 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:25:35.144980 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133426    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:25:35.144980 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133428    5639 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:25:35.144980 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133431    5639 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:25:35.144980 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133433    5639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:25:35.144980 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133435    5639 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:25:35.144980 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133438    5639 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:25:35.144980 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133440    5639 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:25:35.144980 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133444    5639 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:25:35.144980 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133447    5639 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:25:35.144980 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133449    5639 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:25:35.144980 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133451    5639 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:25:35.145608 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133454    5639 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:25:35.145608 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133457    5639 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:25:35.145608 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133459    5639 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:25:35.145608 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133462    5639 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:25:35.145608 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133464    5639 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:25:35.145608 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133468    5639 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:25:35.145608 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133471    5639 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:25:35.145608 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133474    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:25:35.145608 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133477    5639 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:25:35.145608 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133479    5639 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:25:35.145608 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133481    5639 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:25:35.145608 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.133508    5639 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:25:35.145608 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133545    5639 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:25:35.145608 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133550    5639 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:25:35.145608 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133554    5639 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:25:35.145608 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133557    5639 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:25:35.146063 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133560    5639 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:25:35.146063 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133564    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:25:35.146063 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133567    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:25:35.146063 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133569    5639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:25:35.146063 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133572    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:25:35.146063 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133575    5639 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:25:35.146063 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133577    5639 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:25:35.146063 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133580    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:25:35.146063 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133582    5639 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:25:35.146063 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133585    5639 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:25:35.146063 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133587    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:25:35.146063 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133590    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:25:35.146063 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133594    5639 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:25:35.146063 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133596    5639 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:25:35.146063 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133599    5639 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:25:35.146063 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133602    5639 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:25:35.146063 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133604    5639 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:25:35.146063 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133606    5639 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:25:35.146063 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133609    5639 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:25:35.146063 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133612    5639 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:25:35.146817 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133614    5639 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:25:35.146817 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133617    5639 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:25:35.146817 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133620    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:25:35.146817 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133622    5639 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:25:35.146817 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133625    5639 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:25:35.146817 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133627    5639 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:25:35.146817 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133629    5639 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:25:35.146817 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133633    5639 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:25:35.146817 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133636    5639 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:25:35.146817 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133639    5639 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:25:35.146817 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133643    5639 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:25:35.146817 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133646    5639 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:25:35.146817 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133649    5639 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:25:35.146817 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133653    5639 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:25:35.146817 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133655    5639 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:25:35.146817 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133658    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:25:35.146817 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133660    5639 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:25:35.146817 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133663    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:25:35.146817 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133665    5639 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:25:35.147619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133668    5639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:25:35.147619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133670    5639 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:25:35.147619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133672    5639 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:25:35.147619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133675    5639 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:25:35.147619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133677    5639 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:25:35.147619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133681    5639 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:25:35.147619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133684    5639 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:25:35.147619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133686    5639 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:25:35.147619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133689    5639 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:25:35.147619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133691    5639 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:25:35.147619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133694    5639 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:25:35.147619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133696    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:25:35.147619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133699    5639 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:25:35.147619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133701    5639 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:25:35.147619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133704    5639 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:25:35.147619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133706    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:25:35.147619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133708    5639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:25:35.147619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133711    5639 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:25:35.147619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133714    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:25:35.148343 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133717    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:25:35.148343 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133719    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:25:35.148343 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133722    5639 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:25:35.148343 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133724    5639 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:25:35.148343 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133727    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:25:35.148343 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133729    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:25:35.148343 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133732    5639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:25:35.148343 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133736    5639 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:25:35.148343 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133738    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:25:35.148343 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133741    5639 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:25:35.148343 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133743    5639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:25:35.148343 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133746    5639 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:25:35.148343 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133748    5639 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:25:35.148343 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133751    5639 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:25:35.148343 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133753    5639 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:25:35.148343 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133756    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:25:35.148343 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133758    5639 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:25:35.148343 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133761    5639 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:25:35.148343 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133764    5639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:25:35.148343 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133767    5639 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:25:35.148841 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133769    5639 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:25:35.148841 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133772    5639 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:25:35.148841 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133774    5639 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:25:35.148841 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133777    5639 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:25:35.148841 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133779    5639 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:25:35.148841 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133782    5639 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:25:35.148841 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133784    5639 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:25:35.148841 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133787    5639 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:25:35.148841 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133789    5639 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:25:35.148841 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133792    5639 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:25:35.148841 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133794    5639 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:25:35.148841 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133798    5639 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:25:35.148841 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133802    5639 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:25:35.148841 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133805    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:25:35.148841 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133808    5639 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:25:35.148841 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133810    5639 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:25:35.148841 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133813    5639 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:25:35.148841 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133816    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:25:35.148841 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133818    5639 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:25:35.149339 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133823    5639 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:25:35.149339 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133826    5639 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:25:35.149339 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133829    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:25:35.149339 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133831    5639 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:25:35.149339 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.133834    5639 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:25:35.149339 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.137819    5639 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 01:25:35.149339 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.137835    5639 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 01:25:35.149339 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.137854    5639 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 01:25:35.149339 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.137859    5639 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 01:25:35.149339 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137920    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:25:35.149339 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137925    5639 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:25:35.149339 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137929    5639 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:25:35.149339 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137932    5639 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:25:35.149339 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137935    5639 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:25:35.149339 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137938    5639 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:25:35.149339 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137941    5639 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:25:35.149339 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137944    5639 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:25:35.149339 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137947    5639 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:25:35.149339 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137950    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:25:35.149339 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137953    5639 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:25:35.149819 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137956    5639 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:25:35.149819 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137958    5639 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:25:35.149819 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137961    5639 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:25:35.149819 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137963    5639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:25:35.149819 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137966    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:25:35.149819 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137969    5639 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:25:35.149819 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137971    5639 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:25:35.149819 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137974    5639 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:25:35.149819 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137977    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:25:35.149819 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137979    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:25:35.149819 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137982    5639 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:25:35.149819 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137984    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:25:35.149819 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137987    5639 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:25:35.149819 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137990    5639 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:25:35.149819 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137993    5639 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:25:35.149819 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137995    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:25:35.149819 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.137999    5639 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:25:35.149819 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138001    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:25:35.149819 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138004    5639 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:25:35.149819 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138006    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:25:35.150313 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138009    5639 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:25:35.150313 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138012    5639 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:25:35.150313 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138014    5639 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:25:35.150313 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138017    5639 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:25:35.150313 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138019    5639 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:25:35.150313 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138022    5639 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:25:35.150313 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138024    5639 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:25:35.150313 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138027    5639 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:25:35.150313 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138029    5639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:25:35.150313 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138032    5639 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:25:35.150313 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138035    5639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:25:35.150313 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138038    5639 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:25:35.150313 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138041    5639 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:25:35.150313 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138043    5639 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:25:35.150313 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138046    5639 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:25:35.150313 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138048    5639 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:25:35.150313 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138051    5639 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:25:35.150313 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138053    5639 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:25:35.150313 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138056    5639 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:25:35.150313 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138058    5639 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:25:35.150801 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138061    5639 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:25:35.150801 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138064    5639 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:25:35.150801 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138067    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:25:35.150801 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138069    5639 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:25:35.150801 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138072    5639 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:25:35.150801 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138076    5639 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:25:35.150801 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138079    5639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:25:35.150801 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138082    5639 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:25:35.150801 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138086    5639 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:25:35.150801 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138088    5639 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:25:35.150801 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138091    5639 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:25:35.150801 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138093    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:25:35.150801 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138096    5639 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:25:35.150801 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138099    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:25:35.150801 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138101    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:25:35.150801 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138104    5639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:25:35.150801 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138106    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:25:35.150801 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138109    5639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:25:35.150801 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138111    5639 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:25:35.151262 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138114    5639 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:25:35.151262 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138116    5639 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:25:35.151262 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138120    5639 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:25:35.151262 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138124    5639 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:25:35.151262 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138127    5639 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:25:35.151262 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138130    5639 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:25:35.151262 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138133    5639 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:25:35.151262 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138136    5639 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:25:35.151262 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138138    5639 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:25:35.151262 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138141    5639 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:25:35.151262 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138144    5639 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:25:35.151262 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138146    5639 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:25:35.151262 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138148    5639 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:25:35.151262 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138151    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:25:35.151262 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138153    5639 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:25:35.151262 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138156    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:25:35.151262 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138158    5639 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:25:35.151262 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138161    5639 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:25:35.151262 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138164    5639 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:25:35.151844 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138168    5639 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:25:35.151844 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138170    5639 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:25:35.151844 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138173    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:25:35.151844 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138175    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:25:35.151844 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138178    5639 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:25:35.151844 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138180    5639 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:25:35.151844 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138183    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:25:35.151844 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138185    5639 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:25:35.151844 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138188    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:25:35.151844 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138190    5639 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:25:35.151844 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138192    5639 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:25:35.151844 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138195    5639 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:25:35.151844 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138197    5639 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:25:35.151844 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138200    5639 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:25:35.151844 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138202    5639 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:25:35.151844 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138204    5639 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:25:35.151844 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138207    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:25:35.152254 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.138234    5639 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:25:35.152254 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138408    5639 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:25:35.152254 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138414    5639 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:25:35.152254 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138418    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:25:35.152254 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138421    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:25:35.152254 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138423    5639 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:25:35.152254 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138426    5639 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:25:35.152254 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138429    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:25:35.152254 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138432    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:25:35.152254 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138436    5639 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:25:35.152254 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138438    5639 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:25:35.152254 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138441    5639 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:25:35.152254 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138444    5639 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:25:35.152254 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138446    5639 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:25:35.152254 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138449    5639 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:25:35.152254 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138451    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:25:35.152662 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138454    5639 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:25:35.152662 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138457    5639 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:25:35.152662 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138460    5639 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:25:35.152662 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138463    5639 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:25:35.152662 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138465    5639 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:25:35.152662 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138468    5639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:25:35.152662 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138471    5639 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:25:35.152662 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138473    5639 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:25:35.152662 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138476    5639 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:25:35.152662 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138478    5639 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:25:35.152662 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138481    5639 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:25:35.152662 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138484    5639 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:25:35.152662 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138486    5639 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:25:35.152662 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138489    5639 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:25:35.152662 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138492    5639 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:25:35.152662 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138494    5639 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:25:35.152662 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138496    5639 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:25:35.152662 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138499    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:25:35.152662 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138502    5639 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:25:35.152662 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138505    5639 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:25:35.153143 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138507    5639 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:25:35.153143 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138509    5639 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:25:35.153143 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138512    5639 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:25:35.153143 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138515    5639 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:25:35.153143 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138518    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:25:35.153143 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138520    5639 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:25:35.153143 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138523    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:25:35.153143 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138525    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:25:35.153143 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138528    5639 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:25:35.153143 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138532    5639 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:25:35.153143 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138536    5639 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:25:35.153143 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138539    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:25:35.153143 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138543    5639 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:25:35.153143 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138548    5639 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:25:35.153143 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138551    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:25:35.153143 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138554    5639 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:25:35.153143 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138556    5639 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:25:35.153143 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138559    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:25:35.153610 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138561    5639 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:25:35.153610 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138563    5639 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:25:35.153610 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138566    5639 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:25:35.153610 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138568    5639 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:25:35.153610 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138571    5639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:25:35.153610 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138573    5639 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:25:35.153610 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138576    5639 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:25:35.153610 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138578    5639 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:25:35.153610 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138580    5639 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:25:35.153610 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138583    5639 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:25:35.153610 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138585    5639 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:25:35.153610 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138588    5639 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:25:35.153610 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138590    5639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:25:35.153610 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138593    5639 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:25:35.153610 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138596    5639 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:25:35.153610 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138598    5639 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:25:35.153610 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138601    5639 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:25:35.153610 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138603    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:25:35.153610 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138606    5639 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:25:35.153610 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138608    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:25:35.153610 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138611    5639 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:25:35.154119 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138613    5639 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:25:35.154119 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138616    5639 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:25:35.154119 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138618    5639 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:25:35.154119 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138620    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:25:35.154119 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138623    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:25:35.154119 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138625    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:25:35.154119 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138628    5639 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:25:35.154119 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138631    5639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:25:35.154119 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138633    5639 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:25:35.154119 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138636    5639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:25:35.154119 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138638    5639 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:25:35.154119 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138640    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:25:35.154119 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138643    5639 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:25:35.154119 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138645    5639 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:25:35.154119 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138648    5639 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:25:35.154119 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138650    5639 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:25:35.154119 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138652    5639 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:25:35.154119 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138655    5639 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:25:35.154119 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138657    5639 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:25:35.154119 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138660    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:25:35.154611 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138662    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:25:35.154611 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138665    5639 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:25:35.154611 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138668    5639 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:25:35.154611 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138671    5639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:25:35.154611 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138673    5639 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:25:35.154611 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138676    5639 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:25:35.154611 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138678    5639 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:25:35.154611 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138680    5639 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:25:35.154611 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138683    5639 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:25:35.154611 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138685    5639 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:25:35.154611 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138688    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:25:35.154611 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138690    5639 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:25:35.154611 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.138714    5639 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:25:35.154611 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138746    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:25:35.154611 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138751    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:25:35.154611 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138754    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:25:35.155007 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138756    5639 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:25:35.155007 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138760    5639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:25:35.155007 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138762    5639 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:25:35.155007 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138764    5639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:25:35.155007 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138767    5639 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:25:35.155007 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138770    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:25:35.155007 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138772    5639 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:25:35.155007 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138775    5639 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:25:35.155007 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138778    5639 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:25:35.155007 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138780    5639 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:25:35.155007 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138782    5639 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:25:35.155007 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138785    5639 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:25:35.155007 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138787    5639 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:25:35.155007 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138789    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:25:35.155007 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138792    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:25:35.155007 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138795    5639 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:25:35.155007 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138797    5639 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:25:35.155007 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138799    5639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:25:35.155007 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138802    5639 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:25:35.155007 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138805    5639 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:25:35.155619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138807    5639 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:25:35.155619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138810    5639 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:25:35.155619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138812    5639 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:25:35.155619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138815    5639 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:25:35.155619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138817    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:25:35.155619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138820    5639 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:25:35.155619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138822    5639 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:25:35.155619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138825    5639 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:25:35.155619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138827    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:25:35.155619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138830    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:25:35.155619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138833    5639 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:25:35.155619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138835    5639 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:25:35.155619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138838    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:25:35.155619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138841    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:25:35.155619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138843    5639 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:25:35.155619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138846    5639 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:25:35.155619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138848    5639 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:25:35.155619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138851    5639 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:25:35.155619 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138854    5639 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:25:35.156062 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138856    5639 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:25:35.156062 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138858    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:25:35.156062 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138861    5639 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:25:35.156062 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138863    5639 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:25:35.156062 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138866    5639 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:25:35.156062 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138868    5639 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:25:35.156062 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138871    5639 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:25:35.156062 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138873    5639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:25:35.156062 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138876    5639 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:25:35.156062 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138878    5639 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:25:35.156062 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138881    5639 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:25:35.156062 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138883    5639 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:25:35.156062 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138886    5639 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:25:35.156062 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138888    5639 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:25:35.156062 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138891    5639 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:25:35.156062 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138893    5639 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:25:35.156062 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138895    5639 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:25:35.156062 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138898    5639 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:25:35.156062 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138900    5639 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:25:35.156062 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138903    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:25:35.156634 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138905    5639 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:25:35.156634 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138908    5639 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:25:35.156634 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138911    5639 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:25:35.156634 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138913    5639 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:25:35.156634 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138917    5639 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:25:35.156634 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138919    5639 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:25:35.156634 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138922    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:25:35.156634 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138924    5639 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:25:35.156634 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138926    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:25:35.156634 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138929    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:25:35.156634 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138931    5639 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:25:35.156634 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138936    5639 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:25:35.156634 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138939    5639 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:25:35.156634 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138943    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:25:35.156634 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138947    5639 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:25:35.156634 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138950    5639 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:25:35.156634 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138956    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:25:35.156634 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138959    5639 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:25:35.157078 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138961    5639 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:25:35.157078 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138964    5639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:25:35.157078 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138966    5639 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:25:35.157078 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138969    5639 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:25:35.157078 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138971    5639 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:25:35.157078 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138974    5639 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:25:35.157078 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138976    5639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:25:35.157078 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138979    5639 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:25:35.157078 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138981    5639 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:25:35.157078 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138984    5639 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:25:35.157078 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138986    5639 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:25:35.157078 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138989    5639 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:25:35.157078 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138991    5639 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:25:35.157078 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138994    5639 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:25:35.157078 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138996    5639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:25:35.157078 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.138999    5639 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:25:35.157078 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.139001    5639 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:25:35.157078 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.139004    5639 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:25:35.157078 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.139006    5639 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:25:35.157078 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.139009    5639 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:25:35.157078 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.139011    5639 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:25:35.157643 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.139014    5639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:25:35.157643 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.139016    5639 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:25:35.157643 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.139019    5639 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:25:35.157643 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.139021    5639 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:25:35.157643 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:25:35.139024    5639 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:25:35.157643 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.139079    5639 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 01:25:35.157643 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.140060    5639 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 01:25:35.157643 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.140117    5639 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 01:25:35.157643 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.140311    5639 server.go:1014] "Starting client certificate rotation"
Nov 02 01:25:35.157643 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.140469    5639 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 01:25:35.157643 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.140527    5639 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 01:19:24 +0000 UTC" deadline="2028-06-21 16:10:45.242530231 +0000 UTC"
Nov 02 01:25:35.157643 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.140577    5639 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14342h45m10.101956538s"
Nov 02 01:25:35.157643 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.140768    5639 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 01:25:35.157643 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.141967    5639 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 01:25:35.157643 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.143696    5639 log.go:25] "Validated CRI v1 runtime API"
Nov 02 01:25:35.157643 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.145548    5639 log.go:25] "Validated CRI v1 image API"
Nov 02 01:25:35.158030 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.146378    5639 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 01:25:35.158030 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.149489    5639 fs.go:135] Filesystem UUIDs: map[1cc9a6b5-91ef-4870-8a42-6bec1d4eedd9:/dev/nvme0n1p3 30057d81-0a35-49a8-a663-c830ef847dab:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Nov 02 01:25:35.158108 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.149512    5639 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0031722cf46aa740692ee13182e73c8900b973f16aeaef1ea37c75b215e60f6a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0031722cf46aa740692ee13182e73c8900b973f16aeaef1ea37c75b215e60f6a/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0225b89e95550a2d9c8208d8c842473121f6b61a4a17ca74d36c1be1cf61c5e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0225b89e95550a2d9c8208d8c842473121f6b61a4a17ca74d36c1be1cf61c5e2/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0af5a473261fe5bcb54cf153fa5bcb886bc0949c14e8a899c918475a1186efc6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0af5a473261fe5bcb54cf153fa5bcb886bc0949c14e8a899c918475a1186efc6/userdata/shm major:0 minor:276 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/101a936fe9e4dae209c0b3001269273ffeb0cf18a0a57b71d7be1c16b4b4f09d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/101a936fe9e4dae209c0b3001269273ffeb0cf18a0a57b71d7be1c16b4b4f09d/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/20030ecde1b45e8bf123fa19e0c000ce41e91989c2962d24ef6c8f33ef32b5e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/20030ecde1b45e8bf123fa19e0c000ce41e91989c2962d24ef6c8f33ef32b5e1/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/228d085461b0f1841a965aa7a3d32d4ea4a62d7135412a81ba882ffb45fd7a2a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/228d085461b0f1841a965aa7a3d32d4ea4a62d7135412a81ba882ffb45fd7a2a/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31f315b40def8816fd32f0a2dbb100de956ef67ba6cfea10763fefa9102a5503/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31f315b40def8816fd32f0a2dbb100de956ef67ba6cfea10763fefa9102a5503/userdata/shm major:0 minor:327 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3cbbc555c965562205f86366531a5c8550fb5251b3066467acbfdfe46b3be7fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3cbbc555c965562205f86366531a5c8550fb5251b3066467acbfdfe46b3be7fc/userdata/shm major:0 minor:273 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/44577b0e87c6cdea0d415ba1edf6bb51264f55ef8c87add715e7acfb77c1361c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/44577b0e87c6cdea0d415ba1edf6bb51264f55ef8c87add715e7acfb77c1361c/userdata/shm major:0 minor:431 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/577994457ac2d3a23c2897c7646e07c02095ff54922aaae64cd7cacbbfcbffb4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/577994457ac2d3a23c2897c7646e07c02095ff54922aaae64cd7cacbbfcbffb4/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/61cb902d9136cabebfe4c4727597e4cffe7d9ef29d9beb6b2995f98e1e983c87/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/61cb902d9136cabebfe4c4727597e4cffe7d9ef29d9beb6b2995f98e1e983c87/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/687f4030ba6f638575c614f535becdcd181b697bcf2ad0a9ada0c1c0c4333ee7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/687f4030ba6f638575c614f535becdcd181b697bcf2ad0a9ada0c1c0c4333ee7/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/727a3c9aae1952df44a8add24e47b3d47830be1c1ab99729d9895854610cb098/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/727a3c9aae1952df44a8add24e47b3d47830be1c1ab99729d9895854610cb098/userdata/shm major:0 minor:429 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/804900eb52daea113a487364add4a6b182e6bdb792698e117250338b225f0a0c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/804900eb52daea113a487364add4a6b182e6bdb792698e117250338b225f0a0c/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a7840d9f917d98914d9d25551d9543345e78d6cdaa91062c2517ee49ac5782e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a7840d9f917d98914d9d25551d9543345e78d6cdaa91062c2517ee49ac5782e/userdata/shm major:0 minor:355 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/921831ec3fa5204f8af52f0badb8fb2253ac0b7f7bd17aae39a59242775a11f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/921831ec3fa5204f8af52f0badb8fb2253ac0b7f7bd17aae39a59242775a11f2/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94d9247bfa65f5f7793065e3f27f3ecfd7a081eaca2b2e6276d4acf815e12d9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94d9247bfa65f5f7793065e3f27f3ecfd7a081eaca2b2e6276d4acf815e12d9b/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af9eb3b9954bc7ea61ab6821c900840c6b89c8e6a14a0d27ace14b9eaec5138c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af9eb3b9954bc7ea61ab6821c900840c6b89c8e6a14a0d27ace14b9eaec5138c/userdata/shm major:0 minor:434 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c05d8dfb9ad698f6250033cc556920c94cf3ea8930b18ed9c755533bf77b3a81/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c05d8dfb9ad698f6250033cc556920c94cf3ea8930b18ed9c755533bf77b3a81/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5882f07a5330240dcedf16c806e69d7d044aecb03a1c741b2c69a16a5001d8b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5882f07a5330240dcedf16c806e69d7d044aecb03a1c741b2c69a16a5001d8b/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f3f2992af63af9d80f550a68409bda39cd79d1b39871d40123005d198bf3fdfc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f3f2992af63af9d80f550a68409bda39cd79d1b39871d40123005d198bf3fdfc/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7d3e82589ac6e2d16b3a6849e0d70977502e6f899b82854f1d87e695ef7bda8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7d3e82589ac6e2d16b3a6849e0d70977502e6f899b82854f1d87e695ef7bda8/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa4ccb1c824f7e89a7cd55f1b02889b952b1568c13cd4c3d2435ede62d8e1926/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa4ccb1c824f7e89a7cd55f1b02889b952b1568c13cd4c3d2435ede62d8e1926/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23f2f356-b80a-4e8c-a002-c09c006da8a1/volumes/kubernetes.io~projected/kube-api-access-6jd7v:{mountpoint:/var/lib/kubelet/pods/23f2f356-b80a-4e8c-a002-c09c006da8a1/volumes/kubernetes.io~projected/kube-api-access-6jd7v major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23f2f356-b80a-4e8c-a002-c09c006da8a1/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/23f2f356-b80a-4e8c-a002-c09c006da8a1/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:428 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4240f2d9-dd99-4ca9-8447-7706cebcfa08/volumes/kubernetes.io~projected/kube-api-access-p9w59:{mountpoint:/var/lib/kubelet/pods/4240f2d9-dd99-4ca9-8447-7706cebcfa08/volumes/kubernetes.io~projected/kube-api-access-p9w59 major:0 minor:433 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c62a0f0-aefd-44fc-95c1-b2938317ee0d/volumes/kubernetes.io~projected/kube-api-access-xzgv6:{mountpoint:/var/lib/kubelet/pods/4c62a0f0-aefd-44fc-95c1-b2938317ee0d/volumes/kubernetes.io~projected/kube-api-access-xzgv6 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5cc4f303-fe62-44e7-ac7c-719defde064e/volumes/kubernetes.io~projected/kube-api-access-mkznv:{mountpoint:/var/lib/kubelet/pods/5cc4f303-fe62-44e7-ac7c-719defde064e/volumes/kubernetes.io~projected/kube-api-access-mkznv major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5cc4f303-fe62-44e7-ac7c-719defde064e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/5cc4f303-fe62-44e7-ac7c-719defde064e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65dfab40-fb6d-46d7-98f1-8243e194c176/volumes/kubernetes.io~projected/kube-api-access-dbldg:{mountpoint:/var/lib/kubelet/pods/65dfab40-fb6d-46d7-98f1-8243e194c176/volumes/kubernetes.io~projected/kube-api-access-dbldg major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7407afb5-5371-46c7-93ad-896d94a6f015/volumes/kubernetes.io~projected/kube-api-access-9p47d:{mountpoint:/var/lib/kubelet/pods/7407afb5-5371-46c7-93ad-896d94a6f015/volumes/kubernetes.io~projected/kube-api-access-9p47d major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/887b1b53-10f5-4392-89a5-f90a431e60ea/volumes/kubernetes.io~projected/kube-api-access-qfdq8:{mountpoint:/var/lib/kubelet/pods/887b1b53-10f5-4392-89a5-f90a431e60ea/volumes/kubernetes.io~projected/kube-api-access-qfdq8 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/887b1b53-10f5-4392-89a5-f90a431e60ea/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/887b1b53-10f5-4392-89a5-f90a431e60ea/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/887b1b53-10f5-4392-89a5-f90a431e60ea/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/887b1b53-10f5-4392-89a5-f90a431e60ea/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/929a4d2e-02fa-4d5f-ace0-0ad3d41614b7/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/929a4d2e-02fa-4d5f-ace0-0ad3d41614b7/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9eb605c8-f995-49ae-a1b8-06aa55160b13/volumes/kubernetes.io~projected/kube-api-access-zdxpz:{mountpoint:/var/lib/kubelet/pods/9eb605c8-f995-49ae-a1b8-06aa55160b13/volumes/kubernetes.io~projected/kube-api-access-zdxpz major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9eb605c8-f995-49ae-a1b8-06aa55160b13/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/9eb605c8-f995-49ae-a1b8-06aa55160b13/volumes/kubernetes.io~secret/metrics-certs major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac453c3c-9209-4c20-8fe7-07c6f561b855/volumes/kubernetes.io~projected/kube-api-access-ft2nz:{mountpoint:/var/lib/kubelet/pods/ac453c3c-9209-4c20-8fe7-07c6f561b855/volumes/kubernetes.io~projected/kube-api-access-ft2nz major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac453c3c-9209-4c20-8fe7-07c6f561b855/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ac453c3c-9209-4c20-8fe7-07c6f561b855/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac453c3c-9209-4c20-8fe7-07c6f561b855/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/ac453c3c-9209-4c20-8fe7-07c6f561b855/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0d87141-a1da-4d64-b01e-e66d8ba5e9ac/volumes/kubernetes.io~projected/kube-api-access-9fswd:{mountpoint:/var/lib/kubelet/pods/b0d87141-a1da-4d64-b01e-e66d8ba5e9ac/volumes/kubernetes.io~projected/kube-api-access-9fswd major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volume-subpaths/web-config/alertmanager/9 major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volume-subpaths/web-config/config-reloader/6 major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~empty-dir/config-out major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~projected/kube-api-access-bdlxn:{mountpoint:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~projected/kube-api-access-bdlxn major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~projected/tls-assets major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~secret/config-volume major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~secret/web-config major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b49b523f-d706-4d6c-aac0-020b2e75c88a/volumes/kubernetes.io~projected/kube-api-access-jzlrg:{mountpoint:/var/lib/kubelet/pods/b49b523f-d706-4d6c-aac0-020b2e75c88a/volumes/kubernetes.io~projected/kube-api-access-jzlrg major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b49b523f-d706-4d6c-aac0-020b2e75c88a/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/b49b523f-d706-4d6c-aac0-020b2e75c88a/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b49b523f-d706-4d6c-aac0-020b2e75c88a/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/b49b523f-d706-4d6c-aac0-020b2e75c88a/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b49b523f-d706-4d6c-aac0-020b2e75c88a/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/b49b523f-d706-4d6c-aac0-020b2e75c88a/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6238ba5-f475-4e61-94d8-7d7bc13b2458/volumes/kubernetes.io~projected/kube-api-access-zwct4:{mountpoint:/var/lib/kubelet/pods/b6238ba5-f475-4e61-94d8-7d7bc13b2458/volumes/kubernetes.io~projected/kube-api-access-zwct4 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc6acc82-57d4-4906-9c57-67d717e89ba3/volumes/kubernetes.io~projected/kube-api-access-c2n6b:{mountpoint:/var/lib/kubelet/pods/bc6acc82-57d4-4906-9c57-67d717e89ba3/volumes/kubernetes.io~projected/kube-api-access-c2n6b major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc6acc82-57d4-4906-9c57-67d717e89ba3/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/bc6acc82-57d4-4906-9c57-67d717e89ba3/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc6acc82-57d4-4906-9c57-67d717e89ba3/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/bc6acc82-57d4-4906-9c57-67d717e89ba3/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9256ae6-c79a-46f6-b6bd-8176b8907045/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/c9256ae6-c79a-46f6-b6bd-8176b8907045/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9256ae6-c79a-46f6-b6bd-8176b8907045/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/c9256ae6-c79a-46f6-b6bd-8176b8907045/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9256ae6-c79a-46f6-b6bd-8176b8907045/volumes/kubernetes.io~projected/kube-api-access-nsgjl:{mountpoint:/var/lib/kubelet/pods/c9256ae6-c79a-46f6-b6bd-8176b8907045/volumes/kubernetes.io~projected/kube-api-access-nsgjl major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d352c5c1-44a0-4929-a43a-30c723958ce7/volumes/kubernetes.io~projected/kube-api-access-mrkz9:{mountpoint:/var/lib/kubelet/pods/d352c5c1-44a0-4929-a43a-30c723958ce7/volumes/kubernetes.io~projected/kube-api-access-mrkz9 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa/volumes/kubernetes.io~projected/kube-api-access-dck75:{mountpoint:/var/lib/kubelet/pods/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa/volumes/kubernetes.io~projected/kube-api-access-dck75 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de6949bb-b875-4ca6-b0d3-19c07290bfae/volumes/kubernetes.io~projected/kube-api-access-jnlcv:{mountpoint:/var/lib/kubelet/pods/de6949bb-b875-4ca6-b0d3-19c07290bfae/volumes/kubernetes.io~projected/kube-api-access-jnlcv major:0 minor:267 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de6949bb-b875-4ca6-b0d3-19c07290bfae/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/de6949bb-b875-4ca6-b0d3-19c07290bfae/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de6949bb-b875-4ca6-b0d3-19c07290bfae/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/de6949bb-b875-4ca6-b0d3-19c07290bfae/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef8451f5-e54e-42e3-aad7-16e13fb70d81/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/ef8451f5-e54e-42e3-aad7-16e13fb70d81/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef8451f5-e54e-42e3-aad7-16e13fb70d81/volumes/kubernetes.io~projected/kube-api-access-gj9pc:{mountpoint:/var/lib/kubelet/pods/ef8451f5-e54e-42e3-aad7-16e13fb70d81/volumes/kubernetes.io~projected/kube-api-access-gj9pc major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef8451f5-e54e-42e3-aad7-16e13fb70d81/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/ef8451f5-e54e-42e3-aad7-16e13fb70d81/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/volumes/kubernetes.io~projected/kube-api-access-pp2pv:{mountpoint:/var/lib/kubelet/pods/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/volumes/kubernetes.io~projected/kube-api-access-pp2pv major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:335 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/d8409bc5c61f5396f7190d511d3d65c9c8b86ef4e634a9ecc8649922411e122d/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/a525e9647b4be3fa45276dccc53a15f5ba0c6086f1a8371d46a7352cac4dfa00/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/e91cbf12e8d583568520627765dfee3794824c54cef179a7f097cf894ecb79da/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/1a3c8875c076662b1be87b337c7da6358efbba022ca1de32a6520d718cc5cd32/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/ff8bf6220ffd734a0f10c05935c9f3422532a6dc3901f5c70cd6bc3b0e038247/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/cee2df04679eda540d74d6c916f96c899d02248c2f3e749615b657efa1fff06c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/5f189014e2c483fe58d7d9d3f10c5a57ba9889cfd8b792cdbce4d0e172ca76a1/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/365b8af69fcea323c39d58e88763a6f103cd3933a973fa05c57f5c590c579ce5/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/7f5d2153816c07a44b443e0b1478294aef3bf40098e0095fe4677fd20229bf8e/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/da678b05152f3232f4a921d5343f320e2924dcdd0781e2cf7a385c2c4048cde3/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/82c34e203252cff9f9bbc03a255c46ef659afecd0f5c6b217de152818374ef4e/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/f56c0a752b8cbceb301d8d1013e3748d57736cb890999ec9d2f1349dc38db175/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/8a7277bce2e321527a6a5bdc0723298096f673f396330d58da50e77ec85ee490/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/0bf516e436f4db37b5721887ab190290d407d66ed46537b42c3b41ca7745a624/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/721eee1dad2e277a4e3e527e75773476d915cbf37e261ae791202221f92d3748/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-163:{mountpoint:/var/lib/containers/storage/overlay/99600cdb2f36b2bc178ed97f57876b766ada01c01f411edba454e21d646ac485/merged major:0 minor:163 fsType:overlay blockSize:0} overlay_0-169:{mountpoint:/var/lib/containers/storage/overlay/241ab4709491cf9c547fd57035b6c4a1f77bce3eae437c86887f323f7186aac6/merged major:0 minor:169 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/aed61ab4781adc496e098ec0ca1a230f1b9874cd2425e514452c58af1b389143/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/2942c3b3c8343056067f33dba01886cd1283cde1c2a357ae057fc939c3ff5447/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/5c5169df351a024070da4eb1a318c9bb1c4e1dbd4277873d2b2b8535baad1317/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/bbbc2b171c5b7ac08973e2413bed1f7a90ff43bd3e3aa59ee1a53713f6549931/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/7aac176b4e5aad083965c8d9279200ce812483d1ad8dcc9d2e340e28f45e5dcc/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/a6885669ecb494263d6a0939949fbf886233be1ab7637e5a0b0303b707b8b2df/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/fda633bdde14478b5332a792403526eaf9550ea7d639d5f17bd8a31fcd523296/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/eb870d2ffe496a5361ac70e08b900bd30024460bf49a4ceabc94b6c0f6c5f281/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/b87ec4c406547e7280f9ae3a07fa2a49d131a90b411b46200265c21bd9598ebd/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/3f49c80b02c670d84a1500ef8df5c35d442b97522efa54a2c287ccc73934ebf5/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/38155d5b5338b65a965d8f5af14fd6b9746037492b646039cb2d77f0d67149dc/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/c2edab8988cd567b300ae946dfb7f4d86d6f4320021feb7a95308b9e13c2cf05/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/777b411b395988fcdfdcdb76ffbc4e2e6d0da9278eba3701b52e34da69e09664/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/f001019d2d5b2a8a5d86079759a552e1a8bce980ee7be7645acb45582c745e8d/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/51dfbbb063585f3470df1a96f7b2ab71e30902c92b8dae235cc9e475396b551d/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/2488b1226ab44ddbdba27691450a6e0c967b79e163e793f0d325e5fe8009798b/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/1be82d3242775de44fd17d930b2f66b2302ec333331a73b36abccf45d3b2cdf7/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/572948430a8ba432efa7dfd2d85b5bf27a400ff1b4f91566ee7a1eae3711e65f/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/cf4cc1ec6ae98cc5222c12fb30c34de12755801ae7d29714fbfb726db666d395/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/2beb77909500fca40ebfb6115cac2d972ee97d41713450563b2680a49a5a8e14/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/9d4dec9592ff7116feb6e6d2814eebd2ff61b0ecd7cb692f358137e5504e1937/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/53c81fb4d34d118962fe2a17ec398885ca07fdbeaae0d6eff16c33f422884e72/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/6fbc3cb47f0b715fa725265a686cb7e4186b9b35a5cc1d5beac23936550a7832/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/973983cd1f32aeda870c274d1a5c3315c1a44a4a8cca9d0baa8c3d873757bc22/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/77ff59f0d4e19af2b351b8d1aa71935ad9f8d6b2993fcbaaa7e3bb2ea3e4b30b/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/a7564d8ff63d759415591b426e6ed56286f7e07eb20269ca158d58cdfb8be80b/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/7f24bbce07a52034497d6f6ff9b3af1d29e5f165ac4198ffce51a0d335b24616/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/13afbefc3e87c0a89156e527499f0cf3ee945e9c01fe596ff8a7c47fd948b3ac/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/7292a1e280752166e540d601efe90d7275fbcce4b323f471a8c38ce4694d1bb5/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/ad508b14110e6084dba7c07332626714622cda84b607b716f2099acf3ff086cc/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/cc4adbc3ad102386cf246ab745c3de243ae4a9c52c02423e722fdafb79a9cd6a/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/c54333c0240c2eff92dbd9277582bb8826f7c4d7f24bf51f8bd54efe283095f3/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/ab6a96777f915b9a11b6f78c714ebfde4f10461cd346f6edb33ffaa96e55c9c8/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/ef26ebcae21af83115e0543f6bc560d7860b5ed3a8e5c4f2706952044af3278b/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/cd05c9d6124773bbf1cefe2ba90ef2e827979cffc4503e71ae77675be7ea5e00/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/dac4420aff26cf781a1158121781c4156bce530331d04b09c4c455d38646b10d/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/cc857acf55826808e7b581c1e6f3558552a02abc32688a747352b62ad498d902/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/6ca723b78904447f12c1cc327d978e88cc324927b0bcd2a69d3f946264fceabc/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-403:{mountpoint:/var/lib/containers/storage/overlay/0b920f0c4c2b92a1f094d65221226d61797c6fe3f2a23a6e1480cbbae10113fc/merged major:0 minor:403 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/143f4e5d78962732210ec7f9c3b7521ede2772ce8465323c4929987028e39683/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/434977d13b95e780b647b1a9062d6ca6b543ee7f4d59b4a54eb906e242648d3c/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/c69e2c107ad288a8a48f8fa425eba8a6da1c609ac764461868bd14c60a4014c0/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/9931bae5b99ffe04cefa79a921e375a02babb25b6fb897b979d2a19324f72287/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-438:{mountpoint:/var/lib/containers/storage/overlay/389c9ef1ef1da73b960512a9e599376d0b740fe964baf5e0b8fd0805f5b99afd/merged major:0 minor:438 fsType:overlay blockSize:0} overlay_0-440:{mountpoint:/var/lib/containers/storage/overlay/6462647185a005de75fe12e5d530ec961b0773db03168f58f9249606973177e3/merged major:0 minor:440 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/a258724706aac8c171b574206c9d59102228ce8502d43c93bab41d6e8ca27288/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-444:{mountpoint:/var/lib/containers/storage/overlay/9d24814208a21930b523f7d46421bb0f2f8fbd049ff53b1764404c1df28ce7f8/merged major:0 minor:444 fsType:overlay blockSize:0} overlay_0-450:{mountpoint:/var/lib/containers/storage/overlay/0af230ee8db2c6a51b9d1b9e8014014cad08aad7ae7ae78bc0f6b9a8d6da199e/merged major:0 minor:450 fsType:overlay blockSize:0} overlay_0-455:{mountpoint:/var/lib/containers/storage/overlay/dec1559f732e0a46d148444a9d07c8d784c2ecf8f714efd53bb8be81de0a7615/merged major:0 minor:455 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/d9848ff4c527b4867e441c9c42f0375dcea3127b8e6157f62fa313427a5df202/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/f544355a3f62346d5f28883f07af77048ef276dd1135257915d58fe0f81d1310/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/a16f142e184b409b7f1a94b596a47e4e9e88ac59a211ecd1dfa5fbcadc59b1ba/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/de3a058297e5aeee28199663695c25966fb17a3f03d84412cf0db356796d435d/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/3d8b001d1c43fc47c063588830fc1c8a8f150d5205f509d519423c1ff5c439db/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/84f43d60ca65c2aa665f12f5199a4564e2635621a9c233d9fc37bf73d368aafb/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/5f9d53e95339198b740f22de3bc82d4f7a66ec29c9367ad2dc47ffb22e374eb3/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/e2d22483f4c25dba01840a8715a7402553936c5865ec4ba6408d6fca26d2c1d9/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/b4cddc9fb8dba4d12c3049b43eb50327389815f2b3253ca572fc4c1ba91240d8/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/731a4d9969a35901104885e179b32cba6843cbacf8cd7b2c10f329561c17b90e/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 02 01:25:35.158377 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.157833    5639 manager.go:217] Machine: {Timestamp:2026-11-02 01:25:35.157359006 +0000 UTC m=+0.061041112 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3111970 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2068d50a12027b12b614dba11ef171 SystemUUID:ec2068d5-0a12-027b-12b6-14dba11ef171 BootID:79678c34-a51e-4ad8-94e1-a089f15b6fcf Filesystems:[{Device:/var/lib/kubelet/pods/ac453c3c-9209-4c20-8fe7-07c6f561b855/volumes/kubernetes.io~projected/kube-api-access-ft2nz DeviceMajor:0 DeviceMinor:275 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0af5a473261fe5bcb54cf153fa5bcb886bc0949c14e8a899c918475a1186efc6/userdata/shm DeviceMajor:0 DeviceMinor:276 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-450 DeviceMajor:0 DeviceMinor:450 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0d87141-a1da-4d64-b01e-e66d8ba5e9ac/volumes/kubernetes.io~projected/kube-api-access-9fswd DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/61cb902d9136cabebfe4c4727597e4cffe7d9ef29d9beb6b2995f98e1e983c87/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc6acc82-57d4-4906-9c57-67d717e89ba3/volumes/kubernetes.io~projected/kube-api-access-c2n6b DeviceMajor:0 DeviceMinor:268 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-455 DeviceMajor:0 DeviceMinor:455 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:309 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de6949bb-b875-4ca6-b0d3-19c07290bfae/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:266 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c9256ae6-c79a-46f6-b6bd-8176b8907045/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ef8451f5-e54e-42e3-aad7-16e13fb70d81/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-169 DeviceMajor:0 DeviceMinor:169 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/23f2f356-b80a-4e8c-a002-c09c006da8a1/volumes/kubernetes.io~projected/kube-api-access-6jd7v DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bc6acc82-57d4-4906-9c57-67d717e89ba3/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:265 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:313 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:312 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:332 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b49b523f-d706-4d6c-aac0-020b2e75c88a/volumes/kubernetes.io~projected/kube-api-access-jzlrg DeviceMajor:0 DeviceMinor:354 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:335 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-438 DeviceMajor:0 DeviceMinor:438 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b49b523f-d706-4d6c-aac0-020b2e75c88a/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:353 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5cc4f303-fe62-44e7-ac7c-719defde064e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:331 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a7840d9f917d98914d9d25551d9543345e78d6cdaa91062c2517ee49ac5782e/userdata/shm DeviceMajor:0 DeviceMinor:355 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/20030ecde1b45e8bf123fa19e0c000ce41e91989c2962d24ef6c8f33ef32b5e1/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94d9247bfa65f5f7793065e3f27f3ecfd7a081eaca2b2e6276d4acf815e12d9b/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef8451f5-e54e-42e3-aad7-16e13fb70d81/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:333 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:315 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/727a3c9aae1952df44a8add24e47b3d47830be1c1ab99729d9895854610cb098/userdata/shm DeviceMajor:0 DeviceMinor:429 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/921831ec3fa5204f8af52f0badb8fb2253ac0b7f7bd17aae39a59242775a11f2/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5cc4f303-fe62-44e7-ac7c-719defde064e/volumes/kubernetes.io~projected/kube-api-access-mkznv DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-444 DeviceMajor:0 DeviceMinor:444 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de6949bb-b875-4ca6-b0d3-19c07290bfae/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:264 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:334 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b49b523f-d706-4d6c-aac0-020b2e75c88a/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:351 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ac453c3c-9209-4c20-8fe7-07c6f561b855/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:269 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/228d085461b0f1841a965aa7a3d32d4ea4a62d7135412a81ba882ffb45fd7a2a/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/687f4030ba6f638575c614f535becdcd181b697bcf2ad0a9ada0c1c0c4333ee7/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31f315b40def8816fd32f0a2dbb100de956ef67ba6cfea10763fefa9102a5503/userdata/shm DeviceMajor:0 DeviceMinor:327 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7d3e82589ac6e2d16b3a6849e0d70977502e6f899b82854f1d87e695ef7bda8/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa/volumes/kubernetes.io~projected/kube-api-access-dck75 DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b49b523f-d706-4d6c-aac0-020b2e75c88a/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:352 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af9eb3b9954bc7ea61ab6821c900840c6b89c8e6a14a0d27ace14b9eaec5138c/userdata/shm DeviceMajor:0 DeviceMinor:434 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0225b89e95550a2d9c8208d8c842473121f6b61a4a17ca74d36c1be1cf61c5e2/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/44577b0e87c6cdea0d415ba1edf6bb51264f55ef8c87add715e7acfb77c1361c/userdata/shm DeviceMajor:0 DeviceMinor:431 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c9256ae6-c79a-46f6-b6bd-8176b8907045/volumes/kubernetes.io~projected/kube-api-access-nsgjl DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d352c5c1-44a0-4929-a43a-30c723958ce7/volumes/kubernetes.io~projected/kube-api-access-mrkz9 DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa4ccb1c824f7e89a7cd55f1b02889b952b1568c13cd4c3d2435ede62d8e1926/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/804900eb52daea113a487364add4a6b182e6bdb792698e117250338b225f0a0c/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bc6acc82-57d4-4906-9c57-67d717e89ba3/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:259 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65dfab40-fb6d-46d7-98f1-8243e194c176/volumes/kubernetes.io~projected/kube-api-access-dbldg DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de6949bb-b875-4ca6-b0d3-19c07290bfae/volumes/kubernetes.io~projected/kube-api-access-jnlcv DeviceMajor:0 DeviceMinor:267 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:314 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:315 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/ef8451f5-e54e-42e3-aad7-16e13fb70d81/volumes/kubernetes.io~projected/kube-api-access-gj9pc DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e5882f07a5330240dcedf16c806e69d7d044aecb03a1c741b2c69a16a5001d8b/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ac453c3c-9209-4c20-8fe7-07c6f561b855/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:270 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/101a936fe9e4dae209c0b3001269273ffeb0cf18a0a57b71d7be1c16b4b4f09d/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7407afb5-5371-46c7-93ad-896d94a6f015/volumes/kubernetes.io~projected/kube-api-access-9p47d DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~projected/kube-api-access-bdlxn DeviceMajor:0 DeviceMinor:321 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/887b1b53-10f5-4392-89a5-f90a431e60ea/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:315 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/887b1b53-10f5-4392-89a5-f90a431e60ea/volumes/kubernetes.io~projected/kube-api-access-qfdq8 DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6238ba5-f475-4e61-94d8-7d7bc13b2458/volumes/kubernetes.io~projected/kube-api-access-zwct4 DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:307 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:309 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/887b1b53-10f5-4392-89a5-f90a431e60ea/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:311 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4240f2d9-dd99-4ca9-8447-7706cebcfa08/volumes/kubernetes.io~projected/kube-api-access-p9w59 DeviceMajor:0 DeviceMinor:433 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:310 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:428 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c05d8dfb9ad698f6250033cc556920c94cf3ea8930b18ed9c755533bf77b3a81/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3cbbc555c965562205f86366531a5c8550fb5251b3066467acbfdfe46b3be7fc/userdata/shm DeviceMajor:0 DeviceMinor:273 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:308 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:315 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:330 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/929a4d2e-02fa-4d5f-ace0-0ad3d41614b7/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4c62a0f0-aefd-44fc-95c1-b2938317ee0d/volumes/kubernetes.io~projected/kube-api-access-xzgv6 DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/577994457ac2d3a23c2897c7646e07c02095ff54922aaae64cd7cacbbfcbffb4/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/23f2f356-b80a-4e8c-a002-c09c006da8a1/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-403 DeviceMajor:0 DeviceMinor:403 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9eb605c8-f995-49ae-a1b8-06aa55160b13/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:424 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c9256ae6-c79a-46f6-b6bd-8176b8907045/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9eb605c8-f995-49ae-a1b8-06aa55160b13/volumes/kubernetes.io~projected/kube-api-access-zdxpz DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0031722cf46aa740692ee13182e73c8900b973f16aeaef1ea37c75b215e60f6a/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f3f2992af63af9d80f550a68409bda39cd79d1b39871d40123005d198bf3fdfc/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/volumes/kubernetes.io~projected/kube-api-access-pp2pv DeviceMajor:0 DeviceMinor:341 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-440 DeviceMajor:0 DeviceMinor:440 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-163 DeviceMajor:0 DeviceMinor:163 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:101a936fe9e4dae MacAddress:7a:8f:6d:ea:e8:79 Speed:10000 Mtu:8901} {Name:228d085461b0f18 MacAddress:e6:87:fa:95:60:a9 Speed:10000 Mtu:8901} {Name:31f315b40def881 MacAddress:3a:5d:e6:21:7c:b2 Speed:10000 Mtu:8901} {Name:3cbbc555c965562 MacAddress:be:7b:67:5a:e3:97 Speed:10000 Mtu:8901} {Name:44577b0e87c6cde MacAddress:96:8c:10:98:95:a7 Speed:10000 Mtu:8901} {Name:61cb902d9136cab MacAddress:96:8e:d1:39:15:2c Speed:10000 Mtu:8901} {Name:727a3c9aae1952d MacAddress:4e:22:ed:18:fa:d2 Speed:10000 Mtu:8901} {Name:804900eb52daea1 MacAddress:de:01:2e:c7:8a:13 Speed:10000 Mtu:8901} {Name:8a7840d9f917d98 MacAddress:ca:67:55:4d:64:77 Speed:10000 Mtu:8901} {Name:94d9247bfa65f5f MacAddress:0a:15:a6:10:2e:13 Speed:10000 Mtu:8901} {Name:af9eb3b9954bc7e MacAddress:36:a7:c3:91:33:5f Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:64:df:cd:fb:93 Speed:0 Mtu:9001} {Name:br-int MacAddress:5a:e8:b6:8c:46:d0 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:64:df:cd:fb:93 Speed:0 Mtu:9001} {Name:fa4ccb1c824f7e8 MacAddress:8a:56:d3:96:70:f9 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:ca:a7:37:f8:93:da Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:36:cd:dd:5c:1b:bf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 01:25:35.158377 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.158241    5639 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 01:25:35.158377 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.158310    5639 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 01:25:35.158666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.158638    5639 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 01:25:35.158811 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.158667    5639 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-53.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 01:25:35.158853 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.158814    5639 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 01:25:35.158853 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.158822    5639 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 01:25:35.158853 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.158833    5639 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 01:25:35.158853 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.158853    5639 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 01:25:35.158968 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.158960    5639 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 01:25:35.159142 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.159133    5639 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 01:25:35.159250 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.159241    5639 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 01:25:35.159370 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.159299    5639 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 01:25:35.159370 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.159316    5639 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 01:25:35.159370 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.159328    5639 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 01:25:35.159370 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.159337    5639 kubelet.go:405] "Adding apiserver pod source"
Nov 02 01:25:35.159370 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.159345    5639 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 01:25:35.160238 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.160224    5639 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 01:25:35.160272 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.160240    5639 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 01:25:35.160848 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.160827    5639 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 01:25:35.163542 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.163524    5639 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 01:25:35.163542 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.163539    5639 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 01:25:35.163702 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.163651    5639 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 01:25:35.163903 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.163887    5639 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 01:25:35.163964 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.163928    5639 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 01:25:35.164078 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.164066    5639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 01:25:35.164133 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.164090    5639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 01:25:35.164133 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.164100    5639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 01:25:35.164133 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.164110    5639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 01:25:35.164133 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.164119    5639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 01:25:35.164133 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.164128    5639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 01:25:35.164385 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.164139    5639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 01:25:35.164385 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.164147    5639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 01:25:35.164385 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.164158    5639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 01:25:35.164385 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.164168    5639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 01:25:35.164385 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.164187    5639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 01:25:35.164385 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.164201    5639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 01:25:35.164385 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.164271    5639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 01:25:35.164385 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.164298    5639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 01:25:35.167836 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.167820    5639 server.go:1263] "Started kubelet"
Nov 02 01:25:35.167919 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.167875    5639 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 01:25:35.168524 ip-10-0-136-53 systemd[1]: Started Kubernetes Kubelet.
Nov 02 01:25:35.168694 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.168521    5639 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 01:25:35.172472 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.168868    5639 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 01:25:35.172472 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.169464    5639 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 01:25:35.172628 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.172485    5639 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 01:25:35.174507 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.174485    5639 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 01:25:35.174621 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.174521    5639 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 01:25:35.174689 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.174600    5639 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 01:19:25 +0000 UTC" deadline="2028-07-31 18:20:53.798505888 +0000 UTC"
Nov 02 01:25:35.174689 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.174633    5639 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15304h55m18.623876006s"
Nov 02 01:25:35.174793 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:35.174686    5639 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-53.us-west-1.compute.internal\" not found"
Nov 02 01:25:35.174793 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.174698    5639 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 01:25:35.174793 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.174711    5639 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 01:25:35.174793 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.174739    5639 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 01:25:35.176657 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176619    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d352c5c1-44a0-4929-a43a-30c723958ce7" volumeName="kubernetes.io/configmap/d352c5c1-44a0-4929-a43a-30c723958ce7-serviceca" seLinuxMountContext=""
Nov 02 01:25:35.176657 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176656    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de6949bb-b875-4ca6-b0d3-19c07290bfae" volumeName="kubernetes.io/secret/de6949bb-b875-4ca6-b0d3-19c07290bfae-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 02 01:25:35.176817 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176671    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23f2f356-b80a-4e8c-a002-c09c006da8a1" volumeName="kubernetes.io/secret/23f2f356-b80a-4e8c-a002-c09c006da8a1-cert" seLinuxMountContext=""
Nov 02 01:25:35.176817 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176682    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0d87141-a1da-4d64-b01e-e66d8ba5e9ac" volumeName="kubernetes.io/projected/b0d87141-a1da-4d64-b01e-e66d8ba5e9ac-kube-api-access-9fswd" seLinuxMountContext=""
Nov 02 01:25:35.176817 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176694    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc6acc82-57d4-4906-9c57-67d717e89ba3" volumeName="kubernetes.io/secret/bc6acc82-57d4-4906-9c57-67d717e89ba3-openshift-state-metrics-tls" seLinuxMountContext=""
Nov 02 01:25:35.176817 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176705    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d" volumeName="kubernetes.io/empty-dir/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-tmp" seLinuxMountContext=""
Nov 02 01:25:35.176817 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176717    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c62a0f0-aefd-44fc-95c1-b2938317ee0d" volumeName="kubernetes.io/configmap/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-cni-binary-copy" seLinuxMountContext=""
Nov 02 01:25:35.176817 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176729    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c62a0f0-aefd-44fc-95c1-b2938317ee0d" volumeName="kubernetes.io/projected/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-kube-api-access-xzgv6" seLinuxMountContext=""
Nov 02 01:25:35.176817 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176745    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="887b1b53-10f5-4392-89a5-f90a431e60ea" volumeName="kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-oauth-serving-cert" seLinuxMountContext=""
Nov 02 01:25:35.176817 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176756    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b49b523f-d706-4d6c-aac0-020b2e75c88a" volumeName="kubernetes.io/secret/b49b523f-d706-4d6c-aac0-020b2e75c88a-secret-metrics-server-tls" seLinuxMountContext=""
Nov 02 01:25:35.176817 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176769    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6238ba5-f475-4e61-94d8-7d7bc13b2458" volumeName="kubernetes.io/configmap/b6238ba5-f475-4e61-94d8-7d7bc13b2458-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 01:25:35.176817 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176780    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc6acc82-57d4-4906-9c57-67d717e89ba3" volumeName="kubernetes.io/configmap/bc6acc82-57d4-4906-9c57-67d717e89ba3-metrics-client-ca" seLinuxMountContext=""
Nov 02 01:25:35.176817 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176806    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d" volumeName="kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Nov 02 01:25:35.176817 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176818    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac453c3c-9209-4c20-8fe7-07c6f561b855" volumeName="kubernetes.io/configmap/ac453c3c-9209-4c20-8fe7-07c6f561b855-metrics-client-ca" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176832    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac453c3c-9209-4c20-8fe7-07c6f561b855" volumeName="kubernetes.io/configmap/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176844    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0d87141-a1da-4d64-b01e-e66d8ba5e9ac" volumeName="kubernetes.io/empty-dir/b0d87141-a1da-4d64-b01e-e66d8ba5e9ac-tmp-dir" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176872    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc6acc82-57d4-4906-9c57-67d717e89ba3" volumeName="kubernetes.io/secret/bc6acc82-57d4-4906-9c57-67d717e89ba3-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176885    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa" volumeName="kubernetes.io/empty-dir/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa-tmp-dir" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176899    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef8451f5-e54e-42e3-aad7-16e13fb70d81" volumeName="kubernetes.io/configmap/ef8451f5-e54e-42e3-aad7-16e13fb70d81-env-overrides" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176911    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef8451f5-e54e-42e3-aad7-16e13fb70d81" volumeName="kubernetes.io/projected/ef8451f5-e54e-42e3-aad7-16e13fb70d81-kube-api-access-gj9pc" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176922    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d" volumeName="kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-grpc-tls" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176933    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cc4f303-fe62-44e7-ac7c-719defde064e" volumeName="kubernetes.io/configmap/5cc4f303-fe62-44e7-ac7c-719defde064e-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176947    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9eb605c8-f995-49ae-a1b8-06aa55160b13" volumeName="kubernetes.io/projected/9eb605c8-f995-49ae-a1b8-06aa55160b13-kube-api-access-zdxpz" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176959    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac453c3c-9209-4c20-8fe7-07c6f561b855" volumeName="kubernetes.io/projected/ac453c3c-9209-4c20-8fe7-07c6f561b855-kube-api-access-ft2nz" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176971    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac453c3c-9209-4c20-8fe7-07c6f561b855" volumeName="kubernetes.io/secret/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176984    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b24f6847-62c1-4f87-b320-80327be23ca9" volumeName="kubernetes.io/projected/b24f6847-62c1-4f87-b320-80327be23ca9-tls-assets" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176986    5639 factory.go:55] Registering systemd factory
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.176996    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa" volumeName="kubernetes.io/configmap/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa-config-volume" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177005    5639 factory.go:223] Registration of the systemd container factory successfully
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177008    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65dfab40-fb6d-46d7-98f1-8243e194c176" volumeName="kubernetes.io/projected/65dfab40-fb6d-46d7-98f1-8243e194c176-kube-api-access-dbldg" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177031    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="887b1b53-10f5-4392-89a5-f90a431e60ea" volumeName="kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-service-ca" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177043    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="929a4d2e-02fa-4d5f-ace0-0ad3d41614b7" volumeName="kubernetes.io/configmap/929a4d2e-02fa-4d5f-ace0-0ad3d41614b7-konnectivity-ca" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177054    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b24f6847-62c1-4f87-b320-80327be23ca9" volumeName="kubernetes.io/empty-dir/b24f6847-62c1-4f87-b320-80327be23ca9-alertmanager-main-db" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177068    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b24f6847-62c1-4f87-b320-80327be23ca9" volumeName="kubernetes.io/empty-dir/b24f6847-62c1-4f87-b320-80327be23ca9-config-out" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177092    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b49b523f-d706-4d6c-aac0-020b2e75c88a" volumeName="kubernetes.io/configmap/b49b523f-d706-4d6c-aac0-020b2e75c88a-metrics-server-audit-profiles" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177104    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef8451f5-e54e-42e3-aad7-16e13fb70d81" volumeName="kubernetes.io/configmap/ef8451f5-e54e-42e3-aad7-16e13fb70d81-ovnkube-config" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177117    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="887b1b53-10f5-4392-89a5-f90a431e60ea" volumeName="kubernetes.io/empty-dir/887b1b53-10f5-4392-89a5-f90a431e60ea-tmp" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177129    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="887b1b53-10f5-4392-89a5-f90a431e60ea" volumeName="kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-oauth-config" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177141    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b24f6847-62c1-4f87-b320-80327be23ca9" volumeName="kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-config-volume" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177155    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b49b523f-d706-4d6c-aac0-020b2e75c88a" volumeName="kubernetes.io/secret/b49b523f-d706-4d6c-aac0-020b2e75c88a-secret-metrics-server-client-certs" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177167    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d352c5c1-44a0-4929-a43a-30c723958ce7" volumeName="kubernetes.io/projected/d352c5c1-44a0-4929-a43a-30c723958ce7-kube-api-access-mrkz9" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177179    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de6949bb-b875-4ca6-b0d3-19c07290bfae" volumeName="kubernetes.io/empty-dir/de6949bb-b875-4ca6-b0d3-19c07290bfae-volume-directive-shadow" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177195    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef8451f5-e54e-42e3-aad7-16e13fb70d81" volumeName="kubernetes.io/secret/ef8451f5-e54e-42e3-aad7-16e13fb70d81-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177208    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d" volumeName="kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-tls" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177220    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4240f2d9-dd99-4ca9-8447-7706cebcfa08" volumeName="kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177232    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9eb605c8-f995-49ae-a1b8-06aa55160b13" volumeName="kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177244    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b49b523f-d706-4d6c-aac0-020b2e75c88a" volumeName="kubernetes.io/secret/b49b523f-d706-4d6c-aac0-020b2e75c88a-client-ca-bundle" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177257    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6238ba5-f475-4e61-94d8-7d7bc13b2458" volumeName="kubernetes.io/projected/b6238ba5-f475-4e61-94d8-7d7bc13b2458-kube-api-access-zwct4" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177270    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa" volumeName="kubernetes.io/secret/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa-metrics-tls" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177332    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de6949bb-b875-4ca6-b0d3-19c07290bfae" volumeName="kubernetes.io/configmap/de6949bb-b875-4ca6-b0d3-19c07290bfae-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177344    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef8451f5-e54e-42e3-aad7-16e13fb70d81" volumeName="kubernetes.io/configmap/ef8451f5-e54e-42e3-aad7-16e13fb70d81-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177358    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="929a4d2e-02fa-4d5f-ace0-0ad3d41614b7" volumeName="kubernetes.io/secret/929a4d2e-02fa-4d5f-ace0-0ad3d41614b7-agent-certs" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177371    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6238ba5-f475-4e61-94d8-7d7bc13b2458" volumeName="kubernetes.io/configmap/b6238ba5-f475-4e61-94d8-7d7bc13b2458-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177383    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de6949bb-b875-4ca6-b0d3-19c07290bfae" volumeName="kubernetes.io/configmap/de6949bb-b875-4ca6-b0d3-19c07290bfae-metrics-client-ca" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177397    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26cf53e3-9cf3-4ad5-afe2-5c241f6749ad" volumeName="kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret" seLinuxMountContext=""
Nov 02 01:25:35.177388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177408    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65dfab40-fb6d-46d7-98f1-8243e194c176" volumeName="kubernetes.io/configmap/65dfab40-fb6d-46d7-98f1-8243e194c176-iptables-alerter-script" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177420    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="887b1b53-10f5-4392-89a5-f90a431e60ea" volumeName="kubernetes.io/projected/887b1b53-10f5-4392-89a5-f90a431e60ea-kube-api-access-qfdq8" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177432    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b49b523f-d706-4d6c-aac0-020b2e75c88a" volumeName="kubernetes.io/empty-dir/b49b523f-d706-4d6c-aac0-020b2e75c88a-audit-log" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177454    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d" volumeName="kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177466    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="887b1b53-10f5-4392-89a5-f90a431e60ea" volumeName="kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-console-config" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177305    5639 factory.go:153] Registering CRI-O factory
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177479    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="887b1b53-10f5-4392-89a5-f90a431e60ea" volumeName="kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-serving-cert" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177487    5639 factory.go:223] Registration of the crio container factory successfully
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177492    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b24f6847-62c1-4f87-b320-80327be23ca9" volumeName="kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177503    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc6acc82-57d4-4906-9c57-67d717e89ba3" volumeName="kubernetes.io/projected/bc6acc82-57d4-4906-9c57-67d717e89ba3-kube-api-access-c2n6b" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177514    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9256ae6-c79a-46f6-b6bd-8176b8907045" volumeName="kubernetes.io/empty-dir/c9256ae6-c79a-46f6-b6bd-8176b8907045-tmp" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177526    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9256ae6-c79a-46f6-b6bd-8176b8907045" volumeName="kubernetes.io/empty-dir/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-tuned" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177537    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa" volumeName="kubernetes.io/projected/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa-kube-api-access-dck75" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177543    5639 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177550    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23f2f356-b80a-4e8c-a002-c09c006da8a1" volumeName="kubernetes.io/projected/23f2f356-b80a-4e8c-a002-c09c006da8a1-kube-api-access-6jd7v" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177562    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c62a0f0-aefd-44fc-95c1-b2938317ee0d" volumeName="kubernetes.io/configmap/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-multus-daemon-config" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177585    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cc4f303-fe62-44e7-ac7c-719defde064e" volumeName="kubernetes.io/projected/5cc4f303-fe62-44e7-ac7c-719defde064e-kube-api-access-mkznv" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177590    5639 factory.go:103] Registering Raw factory
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177597    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7407afb5-5371-46c7-93ad-896d94a6f015" volumeName="kubernetes.io/projected/7407afb5-5371-46c7-93ad-896d94a6f015-kube-api-access-9p47d" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177606    5639 manager.go:1199] Started watching for new ooms in manager
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177608    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b24f6847-62c1-4f87-b320-80327be23ca9" volumeName="kubernetes.io/configmap/b24f6847-62c1-4f87-b320-80327be23ca9-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177623    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b49b523f-d706-4d6c-aac0-020b2e75c88a" volumeName="kubernetes.io/projected/b49b523f-d706-4d6c-aac0-020b2e75c88a-kube-api-access-jzlrg" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177636    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de6949bb-b875-4ca6-b0d3-19c07290bfae" volumeName="kubernetes.io/projected/de6949bb-b875-4ca6-b0d3-19c07290bfae-kube-api-access-jnlcv" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177647    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d" volumeName="kubernetes.io/configmap/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-metrics-client-ca" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177660    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b24f6847-62c1-4f87-b320-80327be23ca9" volumeName="kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177672    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de6949bb-b875-4ca6-b0d3-19c07290bfae" volumeName="kubernetes.io/secret/de6949bb-b875-4ca6-b0d3-19c07290bfae-kube-state-metrics-tls" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177685    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cc4f303-fe62-44e7-ac7c-719defde064e" volumeName="kubernetes.io/empty-dir/5cc4f303-fe62-44e7-ac7c-719defde064e-data-volume" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177696    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b24f6847-62c1-4f87-b320-80327be23ca9" volumeName="kubernetes.io/projected/b24f6847-62c1-4f87-b320-80327be23ca9-kube-api-access-bdlxn" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177710    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6238ba5-f475-4e61-94d8-7d7bc13b2458" volumeName="kubernetes.io/configmap/b6238ba5-f475-4e61-94d8-7d7bc13b2458-cni-binary-copy" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177723    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d" volumeName="kubernetes.io/projected/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-kube-api-access-pp2pv" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177736    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d" volumeName="kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177748    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac453c3c-9209-4c20-8fe7-07c6f561b855" volumeName="kubernetes.io/empty-dir/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-textfile" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177759    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9256ae6-c79a-46f6-b6bd-8176b8907045" volumeName="kubernetes.io/projected/c9256ae6-c79a-46f6-b6bd-8176b8907045-kube-api-access-nsgjl" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177770    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b24f6847-62c1-4f87-b320-80327be23ca9" volumeName="kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-cluster-tls-config" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177786    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b24f6847-62c1-4f87-b320-80327be23ca9" volumeName="kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177798    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b24f6847-62c1-4f87-b320-80327be23ca9" volumeName="kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-web-config" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177812    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d" volumeName="kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177826    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cc4f303-fe62-44e7-ac7c-719defde064e" volumeName="kubernetes.io/secret/5cc4f303-fe62-44e7-ac7c-719defde064e-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177841    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac453c3c-9209-4c20-8fe7-07c6f561b855" volumeName="kubernetes.io/secret/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-tls" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177857    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b24f6847-62c1-4f87-b320-80327be23ca9" volumeName="kubernetes.io/configmap/b24f6847-62c1-4f87-b320-80327be23ca9-metrics-client-ca" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177872    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b24f6847-62c1-4f87-b320-80327be23ca9" volumeName="kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177887    5639 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b49b523f-d706-4d6c-aac0-020b2e75c88a" volumeName="kubernetes.io/configmap/b49b523f-d706-4d6c-aac0-020b2e75c88a-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177903    5639 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177911    5639 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 01:25:35.179183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.177946    5639 manager.go:319] Starting recovery of all containers
Nov 02 01:25:35.208734 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.208681    5639 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 01:25:35.210606 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.210577    5639 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 01:25:35.210715 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.210614    5639 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 01:25:35.210715 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.210641    5639 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 01:25:35.210715 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:35.210685    5639 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 01:25:35.220727 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.220675    5639 generic.go:356] "Generic (PLEG): container finished" podID="b24f6847-62c1-4f87-b320-80327be23ca9" containerID="a2f9b1af7a2b352155f273921041bf5082eb862f39ee2a70bd1946b4ae7d4b77" exitCode=0
Nov 02 01:25:35.233751 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.233714    5639 generic.go:356] "Generic (PLEG): container finished" podID="b6238ba5-f475-4e61-94d8-7d7bc13b2458" containerID="90b1f9961e8bc4581df246f8480beea1795285880385d6c153bbc71718348b11" exitCode=0
Nov 02 01:25:35.233751 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.233759    5639 generic.go:356] "Generic (PLEG): container finished" podID="b6238ba5-f475-4e61-94d8-7d7bc13b2458" containerID="8e778cb3addd874bc41c01983d7f8fd9a3799a2eeddf46b392e26720f5b5bc83" exitCode=0
Nov 02 01:25:35.233936 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.233774    5639 generic.go:356] "Generic (PLEG): container finished" podID="b6238ba5-f475-4e61-94d8-7d7bc13b2458" containerID="ea00fa33606ed3cbc2ae2cf2fe3b983d0216f0b3dd023254ec146cffb8598534" exitCode=0
Nov 02 01:25:35.233936 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.233786    5639 generic.go:356] "Generic (PLEG): container finished" podID="b6238ba5-f475-4e61-94d8-7d7bc13b2458" containerID="b7197164ea85fd59c26fc348829cc5f8aab3b8b46960bf850ce86902cac955b8" exitCode=0
Nov 02 01:25:35.233936 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.233799    5639 generic.go:356] "Generic (PLEG): container finished" podID="b6238ba5-f475-4e61-94d8-7d7bc13b2458" containerID="d755b7d37b846d9ff794bf9ada723a11092bacd52ff0900c4b03d1af21c94a27" exitCode=0
Nov 02 01:25:35.233936 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.233808    5639 generic.go:356] "Generic (PLEG): container finished" podID="b6238ba5-f475-4e61-94d8-7d7bc13b2458" containerID="0dcb69b56eb122074d5a6c4212ae6bb997769ee7849913be25f474d2dc5f4f46" exitCode=0
Nov 02 01:25:35.258189 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.258160    5639 generic.go:356] "Generic (PLEG): container finished" podID="ac453c3c-9209-4c20-8fe7-07c6f561b855" containerID="a9bb81938ecc9638a84f5809bcd6a532e79b1a0ca7a1d9164c44b6240a98e934" exitCode=0
Nov 02 01:25:35.265005 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.264972    5639 generic.go:356] "Generic (PLEG): container finished" podID="48935a76d1ff2f699968aa919b673e4d" containerID="effe724ff6b15d0a421751ff5e6d416b9676a987681b1d0c044279c649ece3be" exitCode=0
Nov 02 01:25:35.274777 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:35.274753    5639 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-53.us-west-1.compute.internal\" not found"
Nov 02 01:25:35.292374 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.292314    5639 manager.go:324] Recovery completed
Nov 02 01:25:35.311356 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:35.311321    5639 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 02 01:25:35.313874 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.313859    5639 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:25:35.315761 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.315743    5639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:25:35.315831 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.315781    5639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:25:35.315831 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.315793    5639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:25:35.319245 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.319221    5639 cpu_manager.go:226] "Starting" policy="none"
Nov 02 01:25:35.319245 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.319236    5639 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 01:25:35.319406 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.319259    5639 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 01:25:35.319406 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.319237    5639 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:25:35.319482 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.319452    5639 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 01:25:35.319516 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.319468    5639 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 01:25:35.319516 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.319491    5639 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 01:25:35.319516 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.319499    5639 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 01:25:35.319516 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.319506    5639 policy_none.go:50] "Start"
Nov 02 01:25:35.319633 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.319519    5639 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 01:25:35.319633 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.319533    5639 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 01:25:35.319695 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.319684    5639 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 01:25:35.319724 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.319695    5639 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 01:25:35.319724 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.319701    5639 policy_none.go:44] "Start"
Nov 02 01:25:35.320256 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.320237    5639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:25:35.320355 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.320273    5639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:25:35.320355 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.320307    5639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:25:35.322241 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.322220    5639 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:25:35.322241 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.322221    5639 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:25:35.324650 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.324624    5639 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 01:25:35.327200 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.327186    5639 manager.go:341] "Starting Device Plugin manager"
Nov 02 01:25:35.327265 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:35.327211    5639 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 01:25:35.327265 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.327232    5639 server.go:87] "Starting device plugin registration server"
Nov 02 01:25:35.327601 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.327589    5639 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 01:25:35.328258 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.327605    5639 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 01:25:35.328364 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.328304    5639 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 01:25:35.328445 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.328403    5639 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 01:25:35.328545 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.328447    5639 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 01:25:35.328545 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.328454    5639 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 01:25:35.328633 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.328495    5639 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T01:25:35.328430102Z","UUID":"dc3af2cd-5a30-4734-81b5-0b7ddd9d799b","Handler":null,"Name":"","Endpoint":""}
Nov 02 01:25:35.329526 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:35.329509    5639 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 01:25:35.329612 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:35.329543    5639 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-53.us-west-1.compute.internal\" not found"
Nov 02 01:25:35.329710 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.329692    5639 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 01:25:35.329761 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.329730    5639 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 01:25:35.372009 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.371977    5639 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 01:25:35.420750 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.420724    5639 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:25:35.428471 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.428448    5639 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:25:35.429578 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.429562    5639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:25:35.429637 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.429594    5639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:25:35.429637 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.429606    5639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-53.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:25:35.429715 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.429652    5639 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:25:35.470524 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.470485    5639 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:25:35.512299 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.512235    5639 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal"]
Nov 02 01:25:35.513052 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.513032    5639 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:25:35.513172 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.513079    5639 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:25:35.513172 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.513006    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal" event={"ID":"14b7d72c2c054e228c464e14811dfb58","Type":"ContainerStarted","Data":"5f1ef6b25227e3114be00aa9226a89f4ad7c50eb2abca5c1d5064396520fdcb0"}
Nov 02 01:25:35.513172 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.513149    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal" event={"ID":"14b7d72c2c054e228c464e14811dfb58","Type":"ContainerStarted","Data":"687f4030ba6f638575c614f535becdcd181b697bcf2ad0a9ada0c1c0c4333ee7"}
Nov 02 01:25:35.513340 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.513236    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal" event={"ID":"48935a76d1ff2f699968aa919b673e4d","Type":"ContainerStarted","Data":"b0112150ec9f8bf128d80e5b7d75dd2683457017c04e8a2e5d6a9db82ba58419"}
Nov 02 01:25:35.513340 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.513255    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal" event={"ID":"48935a76d1ff2f699968aa919b673e4d","Type":"ContainerDied","Data":"effe724ff6b15d0a421751ff5e6d416b9676a987681b1d0c044279c649ece3be"}
Nov 02 01:25:35.513340 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.513272    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal" event={"ID":"48935a76d1ff2f699968aa919b673e4d","Type":"ContainerStarted","Data":"577994457ac2d3a23c2897c7646e07c02095ff54922aaae64cd7cacbbfcbffb4"}
Nov 02 01:25:35.539108 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.539080    5639 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:25:35.539227 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.539161    5639 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:25:35.575437 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.575336    5639 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:25:35.575437 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:35.575403    5639 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:25:35.577195 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.577177    5639 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:25:35.577241 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:35.577218    5639 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:25:35.674431 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.674388    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14b7d72c2c054e228c464e14811dfb58-config\") pod \"kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal\" (UID: \"14b7d72c2c054e228c464e14811dfb58\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:25:35.674579 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.674426    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/48935a76d1ff2f699968aa919b673e4d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal\" (UID: \"48935a76d1ff2f699968aa919b673e4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:25:35.674579 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.674467    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48935a76d1ff2f699968aa919b673e4d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal\" (UID: \"48935a76d1ff2f699968aa919b673e4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:25:35.774789 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.774751    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48935a76d1ff2f699968aa919b673e4d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal\" (UID: \"48935a76d1ff2f699968aa919b673e4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:25:35.774943 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.774804    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14b7d72c2c054e228c464e14811dfb58-config\") pod \"kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal\" (UID: \"14b7d72c2c054e228c464e14811dfb58\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:25:35.774943 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.774825    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/48935a76d1ff2f699968aa919b673e4d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal\" (UID: \"48935a76d1ff2f699968aa919b673e4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:25:35.774943 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.774756    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48935a76d1ff2f699968aa919b673e4d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal\" (UID: \"48935a76d1ff2f699968aa919b673e4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:25:35.774943 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.774865    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/48935a76d1ff2f699968aa919b673e4d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal\" (UID: \"48935a76d1ff2f699968aa919b673e4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:25:35.774943 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:35.774884    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14b7d72c2c054e228c464e14811dfb58-config\") pod \"kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal\" (UID: \"14b7d72c2c054e228c464e14811dfb58\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:25:36.159525 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.159483    5639 apiserver.go:52] "Watching apiserver"
Nov 02 01:25:36.263572 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.263541    5639 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 01:25:36.264185 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.264164    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-mjlgj","openshift-insights/insights-runtime-extractor-vrpsb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal","openshift-monitoring/metrics-server-5bfddd89bf-5vgk2","openshift-monitoring/alertmanager-main-0","kube-system/konnectivity-agent-vlg9k","openshift-console/console-86d9455949-gsxx6","openshift-ingress-canary/ingress-canary-bncw7","openshift-image-registry/node-ca-9lvcx","openshift-monitoring/thanos-querier-c68bc9867-x88cx","kube-system/global-pull-secret-syncer-9kqsh","kube-system/kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978","openshift-monitoring/openshift-state-metrics-7bd7489957-cj277","openshift-dns/node-resolver-bflth","openshift-monitoring/node-exporter-s59gt","openshift-ovn-kubernetes/ovnkube-node-4vjq9","openshift-dns/dns-default-mjwbd","openshift-multus/network-metrics-daemon-rfwwn","openshift-network-diagnostics/network-check-target-w4jf8","openshift-cluster-node-tuning-operator/tuned-stb4n","openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f","openshift-multus/multus-additional-cni-plugins-kmzfl","openshift-multus/multus-n5rzm"]
Nov 02 01:25:36.277529 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.277506    5639 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:25:36.277861 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.277722    5639 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:25:36.278186 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.278120    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-run-multus-certs\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.278360 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.278194    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/887b1b53-10f5-4392-89a5-f90a431e60ea-tmp\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:36.278781 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.278755    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:36.278881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.278787    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9256ae6-c79a-46f6-b6bd-8176b8907045-tmp\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.278881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.278759    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/887b1b53-10f5-4392-89a5-f90a431e60ea-tmp\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:36.278881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.278806    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-var-lib-cni-multus\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.278881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.278848    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-oauth-serving-cert\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:36.279150 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.278890    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-run-netns\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.279150 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.278907    5639 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 02 01:25:36.279150 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.278909    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9256ae6-c79a-46f6-b6bd-8176b8907045-tmp\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.279150 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.278985    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-cni-bin\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.279150 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279021    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/de6949bb-b875-4ca6-b0d3-19c07290bfae-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:36.279150 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.279048    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-oauth-serving-cert podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:36.77902877 +0000 UTC m=+1.682710862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-oauth-serving-cert") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 02 01:25:36.279150 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279063    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5cc4f303-fe62-44e7-ac7c-719defde064e-crio-socket\") pod \"insights-runtime-extractor-vrpsb\" (UID: \"5cc4f303-fe62-44e7-ac7c-719defde064e\") " pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:36.279150 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279111    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-cnibin\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.279150 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279112    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/de6949bb-b875-4ca6-b0d3-19c07290bfae-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:36.279622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279139    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa-tmp-dir\") pod \"dns-default-mjwbd\" (UID: \"dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa\") " pod="openshift-dns/dns-default-mjwbd"
Nov 02 01:25:36.279622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279178    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-serving-cert\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:36.279622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279197    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-modprobe-d\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.279622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279220    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ef8451f5-e54e-42e3-aad7-16e13fb70d81-ovn-node-metrics-cert\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.279622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279261    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa-tmp-dir\") pod \"dns-default-mjwbd\" (UID: \"dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa\") " pod="openshift-dns/dns-default-mjwbd"
Nov 02 01:25:36.279622 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.279296    5639 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 02 01:25:36.279622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279320    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/de6949bb-b875-4ca6-b0d3-19c07290bfae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:36.279622 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.279337    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-serving-cert podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:36.779328342 +0000 UTC m=+1.683010434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-serving-cert") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"console-serving-cert" not registered
Nov 02 01:25:36.279622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279354    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs\") pod \"network-metrics-daemon-rfwwn\" (UID: \"9eb605c8-f995-49ae-a1b8-06aa55160b13\") " pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:25:36.279622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279369    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b0d87141-a1da-4d64-b01e-e66d8ba5e9ac-hosts-file\") pod \"node-resolver-bflth\" (UID: \"b0d87141-a1da-4d64-b01e-e66d8ba5e9ac\") " pod="openshift-dns/node-resolver-bflth"
Nov 02 01:25:36.279622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279383    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6238ba5-f475-4e61-94d8-7d7bc13b2458-cnibin\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:25:36.279622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279446    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de6949bb-b875-4ca6-b0d3-19c07290bfae-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:36.279622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279471    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-sysconfig\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.279622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279486    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-textfile\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:36.279622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279522    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-var-lib-kubelet\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.279622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279553    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-textfile\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:36.279622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279563    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-socket-dir\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:25:36.279622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279582    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-oauth-config\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:36.279622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279614    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6238ba5-f475-4e61-94d8-7d7bc13b2458-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:25:36.279622 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.279634    5639 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 02 01:25:36.279622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279639    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-hostroot\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279654    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-cni-netd\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.279687    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-oauth-config podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:36.779671091 +0000 UTC m=+1.683353201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-oauth-config") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"console-oauth-config" not registered
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279704    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-host\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279737    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-system-cni-dir\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279765    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/929a4d2e-02fa-4d5f-ace0-0ad3d41614b7-agent-certs\") pod \"konnectivity-agent-vlg9k\" (UID: \"929a4d2e-02fa-4d5f-ace0-0ad3d41614b7\") " pod="kube-system/konnectivity-agent-vlg9k"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279781    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbldg\" (UniqueName: \"kubernetes.io/projected/65dfab40-fb6d-46d7-98f1-8243e194c176-kube-api-access-dbldg\") pod \"iptables-alerter-mjlgj\" (UID: \"65dfab40-fb6d-46d7-98f1-8243e194c176\") " pod="openshift-network-operator/iptables-alerter-mjlgj"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279860    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-accelerators-collector-config\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279880    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnlcv\" (UniqueName: \"kubernetes.io/projected/de6949bb-b875-4ca6-b0d3-19c07290bfae-kube-api-access-jnlcv\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279894    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzgv6\" (UniqueName: \"kubernetes.io/projected/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-kube-api-access-xzgv6\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279910    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-run-ovn-kubernetes\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279925    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fswd\" (UniqueName: \"kubernetes.io/projected/b0d87141-a1da-4d64-b01e-e66d8ba5e9ac-kube-api-access-9fswd\") pod \"node-resolver-bflth\" (UID: \"b0d87141-a1da-4d64-b01e-e66d8ba5e9ac\") " pod="openshift-dns/node-resolver-bflth"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.279939    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-console-config\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.279986    5639 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.280022    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-console-config podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:36.78001274 +0000 UTC m=+1.683694833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-console-config") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"console-config" not registered
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280057    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc6acc82-57d4-4906-9c57-67d717e89ba3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-cj277\" (UID: \"bc6acc82-57d4-4906-9c57-67d717e89ba3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280128    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-var-lib-cni-bin\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280147    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6238ba5-f475-4e61-94d8-7d7bc13b2458-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280167    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ef8451f5-e54e-42e3-aad7-16e13fb70d81-env-overrides\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280212    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-registration-dir\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280267    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-cni-binary-copy\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280343    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-run-netns\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280374    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa-metrics-tls\") pod \"dns-default-mjwbd\" (UID: \"dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa\") " pod="openshift-dns/dns-default-mjwbd"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280396    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrkz9\" (UniqueName: \"kubernetes.io/projected/d352c5c1-44a0-4929-a43a-30c723958ce7-kube-api-access-mrkz9\") pod \"node-ca-9lvcx\" (UID: \"d352c5c1-44a0-4929-a43a-30c723958ce7\") " pod="openshift-image-registry/node-ca-9lvcx"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280412    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6238ba5-f475-4e61-94d8-7d7bc13b2458-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280429    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-device-dir\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280446    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d87141-a1da-4d64-b01e-e66d8ba5e9ac-tmp-dir\") pod \"node-resolver-bflth\" (UID: \"b0d87141-a1da-4d64-b01e-e66d8ba5e9ac\") " pod="openshift-dns/node-resolver-bflth"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280461    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-run-k8s-cni-cncf-io\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280483    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret\") pod \"global-pull-secret-syncer-9kqsh\" (UID: \"26cf53e3-9cf3-4ad5-afe2-5c241f6749ad\") " pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280519    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d352c5c1-44a0-4929-a43a-30c723958ce7-host\") pod \"node-ca-9lvcx\" (UID: \"d352c5c1-44a0-4929-a43a-30c723958ce7\") " pod="openshift-image-registry/node-ca-9lvcx"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280544    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5cc4f303-fe62-44e7-ac7c-719defde064e-data-volume\") pod \"insights-runtime-extractor-vrpsb\" (UID: \"5cc4f303-fe62-44e7-ac7c-719defde064e\") " pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280575    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-run-ovn\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280588    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d87141-a1da-4d64-b01e-e66d8ba5e9ac-tmp-dir\") pod \"node-resolver-bflth\" (UID: \"b0d87141-a1da-4d64-b01e-e66d8ba5e9ac\") " pod="openshift-dns/node-resolver-bflth"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280633    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5cc4f303-fe62-44e7-ac7c-719defde064e-data-volume\") pod \"insights-runtime-extractor-vrpsb\" (UID: \"5cc4f303-fe62-44e7-ac7c-719defde064e\") " pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:36.280666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280661    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-etc-openvswitch\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280691    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-multus-socket-dir-parent\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280740    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-kubelet-config\") pod \"global-pull-secret-syncer-9kqsh\" (UID: \"26cf53e3-9cf3-4ad5-afe2-5c241f6749ad\") " pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280765    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-wtmp\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280782    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-multus-cni-dir\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280818    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-etc-kubernetes\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280841    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-sysctl-d\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280870    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsgjl\" (UniqueName: \"kubernetes.io/projected/c9256ae6-c79a-46f6-b6bd-8176b8907045-kube-api-access-nsgjl\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280901    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/929a4d2e-02fa-4d5f-ace0-0ad3d41614b7-konnectivity-ca\") pod \"konnectivity-agent-vlg9k\" (UID: \"929a4d2e-02fa-4d5f-ace0-0ad3d41614b7\") " pod="kube-system/konnectivity-agent-vlg9k"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280946    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dck75\" (UniqueName: \"kubernetes.io/projected/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa-kube-api-access-dck75\") pod \"dns-default-mjwbd\" (UID: \"dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa\") " pod="openshift-dns/dns-default-mjwbd"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280967    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac453c3c-9209-4c20-8fe7-07c6f561b855-root\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.280997    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-node-log\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281029    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-log-socket\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281064    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ef8451f5-e54e-42e3-aad7-16e13fb70d81-ovnkube-config\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281084    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2n6b\" (UniqueName: \"kubernetes.io/projected/bc6acc82-57d4-4906-9c57-67d717e89ba3-kube-api-access-c2n6b\") pod \"openshift-state-metrics-7bd7489957-cj277\" (UID: \"bc6acc82-57d4-4906-9c57-67d717e89ba3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281109    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-dbus\") pod \"global-pull-secret-syncer-9kqsh\" (UID: \"26cf53e3-9cf3-4ad5-afe2-5c241f6749ad\") " pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281192    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/de6949bb-b875-4ca6-b0d3-19c07290bfae-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281227    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-multus-daemon-config\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281305    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-etc-selinux\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281331    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p47d\" (UniqueName: \"kubernetes.io/projected/7407afb5-5371-46c7-93ad-896d94a6f015-kube-api-access-9p47d\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281347    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jd7v\" (UniqueName: \"kubernetes.io/projected/23f2f356-b80a-4e8c-a002-c09c006da8a1-kube-api-access-6jd7v\") pod \"ingress-canary-bncw7\" (UID: \"23f2f356-b80a-4e8c-a002-c09c006da8a1\") " pod="openshift-ingress-canary/ingress-canary-bncw7"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281371    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ef8451f5-e54e-42e3-aad7-16e13fb70d81-ovnkube-script-lib\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281399    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-kubernetes\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281440    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-service-ca\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281473    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65dfab40-fb6d-46d7-98f1-8243e194c176-host-slash\") pod \"iptables-alerter-mjlgj\" (UID: \"65dfab40-fb6d-46d7-98f1-8243e194c176\") " pod="openshift-network-operator/iptables-alerter-mjlgj"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.281507    5639 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281506    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac453c3c-9209-4c20-8fe7-07c6f561b855-sys\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281542    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.281579    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-service-ca podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:36.781569297 +0000 UTC m=+1.685251402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-service-ca") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"service-ca" not registered
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281605    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-systemd-units\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281623    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-run-systemd\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281642    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-sys\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281659    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc6acc82-57d4-4906-9c57-67d717e89ba3-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-cj277\" (UID: \"bc6acc82-57d4-4906-9c57-67d717e89ba3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281698    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfdq8\" (UniqueName: \"kubernetes.io/projected/887b1b53-10f5-4392-89a5-f90a431e60ea-kube-api-access-qfdq8\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281722    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9w59\" (UniqueName: \"kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59\") pod \"network-check-target-w4jf8\" (UID: \"4240f2d9-dd99-4ca9-8447-7706cebcfa08\") " pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281806    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-sysctl-conf\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281828    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-systemd\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281842    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-lib-modules\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281869    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5cc4f303-fe62-44e7-ac7c-719defde064e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vrpsb\" (UID: \"5cc4f303-fe62-44e7-ac7c-719defde064e\") " pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281910    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-tls\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281937    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-var-lib-openvswitch\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281954    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-multus-conf-dir\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281970    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65dfab40-fb6d-46d7-98f1-8243e194c176-iptables-alerter-script\") pod \"iptables-alerter-mjlgj\" (UID: \"65dfab40-fb6d-46d7-98f1-8243e194c176\") " pod="openshift-network-operator/iptables-alerter-mjlgj"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.281996    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de6949bb-b875-4ca6-b0d3-19c07290bfae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.282027    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdxpz\" (UniqueName: \"kubernetes.io/projected/9eb605c8-f995-49ae-a1b8-06aa55160b13-kube-api-access-zdxpz\") pod \"network-metrics-daemon-rfwwn\" (UID: \"9eb605c8-f995-49ae-a1b8-06aa55160b13\") " pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.282093    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-var-lib-kubelet\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.282117    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d352c5c1-44a0-4929-a43a-30c723958ce7-serviceca\") pod \"node-ca-9lvcx\" (UID: \"d352c5c1-44a0-4929-a43a-30c723958ce7\") " pod="openshift-image-registry/node-ca-9lvcx"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.282134    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkznv\" (UniqueName: \"kubernetes.io/projected/5cc4f303-fe62-44e7-ac7c-719defde064e-kube-api-access-mkznv\") pod \"insights-runtime-extractor-vrpsb\" (UID: \"5cc4f303-fe62-44e7-ac7c-719defde064e\") " pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.282158    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.282180    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-sys-fs\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.282196    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa-config-volume\") pod \"dns-default-mjwbd\" (UID: \"dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa\") " pod="openshift-dns/dns-default-mjwbd"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.282211    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6238ba5-f475-4e61-94d8-7d7bc13b2458-system-cni-dir\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.282241    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5cc4f303-fe62-44e7-ac7c-719defde064e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vrpsb\" (UID: \"5cc4f303-fe62-44e7-ac7c-719defde064e\") " pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.282293    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23f2f356-b80a-4e8c-a002-c09c006da8a1-cert\") pod \"ingress-canary-bncw7\" (UID: \"23f2f356-b80a-4e8c-a002-c09c006da8a1\") " pod="openshift-ingress-canary/ingress-canary-bncw7"
Nov 02 01:25:36.282263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.282313    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-run-openvswitch\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.284459 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.282343    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac453c3c-9209-4c20-8fe7-07c6f561b855-metrics-client-ca\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:36.284459 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.282376    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft2nz\" (UniqueName: \"kubernetes.io/projected/ac453c3c-9209-4c20-8fe7-07c6f561b855-kube-api-access-ft2nz\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:36.284459 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.282833    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6238ba5-f475-4e61-94d8-7d7bc13b2458-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:25:36.284459 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.282858    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-kubelet\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.284459 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.282874    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj9pc\" (UniqueName: \"kubernetes.io/projected/ef8451f5-e54e-42e3-aad7-16e13fb70d81-kube-api-access-gj9pc\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.284459 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.282895    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-tuned\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.284459 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.282937    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwct4\" (UniqueName: \"kubernetes.io/projected/b6238ba5-f475-4e61-94d8-7d7bc13b2458-kube-api-access-zwct4\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:25:36.284459 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.282967    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-slash\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.284459 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.282971    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-tuned\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.284459 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.282999    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-run\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.284459 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.283025    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc6acc82-57d4-4906-9c57-67d717e89ba3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-cj277\" (UID: \"bc6acc82-57d4-4906-9c57-67d717e89ba3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277"
Nov 02 01:25:36.318989 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.318958    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:25:36.319163 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.319086    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 01:25:36.319163 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.319092    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 01:25:36.319163 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.319128    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 01:25:36.319163 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.319148    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 01:25:36.319345 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.319178    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4gwsn\""
Nov 02 01:25:36.319345 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.319178    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 01:25:36.319345 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.319345    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 01:25:36.319426 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.319349    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 01:25:36.319526 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.319513    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kv2jl\""
Nov 02 01:25:36.319526 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.319514    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-gsg88\""
Nov 02 01:25:36.319623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.319548    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 01:25:36.320526 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.320507    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6238ba5-f475-4e61-94d8-7d7bc13b2458-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:25:36.320594 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.320508    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-cni-binary-copy\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.320724 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.320711    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6238ba5-f475-4e61-94d8-7d7bc13b2458-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:25:36.321862 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.321848    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-multus-daemon-config\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.324310 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.323264    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65dfab40-fb6d-46d7-98f1-8243e194c176-iptables-alerter-script\") pod \"iptables-alerter-mjlgj\" (UID: \"65dfab40-fb6d-46d7-98f1-8243e194c176\") " pod="openshift-network-operator/iptables-alerter-mjlgj"
Nov 02 01:25:36.324310 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.323316    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6238ba5-f475-4e61-94d8-7d7bc13b2458-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:25:36.376822 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.376790    5639 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 01:25:36.381440 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.381418    5639 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 02 01:25:36.381509 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.381447    5639 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 02 01:25:36.381509 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.381458    5639 projected.go:196] Error preparing data for projected volume kube-api-access-qfdq8 for pod openshift-console/console-86d9455949-gsxx6: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 02 01:25:36.381585 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.381530    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/887b1b53-10f5-4392-89a5-f90a431e60ea-kube-api-access-qfdq8 podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:36.881514555 +0000 UTC m=+1.785196646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qfdq8" (UniqueName: "kubernetes.io/projected/887b1b53-10f5-4392-89a5-f90a431e60ea-kube-api-access-qfdq8") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 02 01:25:36.381921 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.381897    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdxpz\" (UniqueName: \"kubernetes.io/projected/9eb605c8-f995-49ae-a1b8-06aa55160b13-kube-api-access-zdxpz\") pod \"network-metrics-daemon-rfwwn\" (UID: \"9eb605c8-f995-49ae-a1b8-06aa55160b13\") " pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:25:36.381958 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.381919    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbldg\" (UniqueName: \"kubernetes.io/projected/65dfab40-fb6d-46d7-98f1-8243e194c176-kube-api-access-dbldg\") pod \"iptables-alerter-mjlgj\" (UID: \"65dfab40-fb6d-46d7-98f1-8243e194c176\") " pod="openshift-network-operator/iptables-alerter-mjlgj"
Nov 02 01:25:36.383848 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.383829    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65dfab40-fb6d-46d7-98f1-8243e194c176-host-slash\") pod \"iptables-alerter-mjlgj\" (UID: \"65dfab40-fb6d-46d7-98f1-8243e194c176\") " pod="openshift-network-operator/iptables-alerter-mjlgj"
Nov 02 01:25:36.383920 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.383855    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac453c3c-9209-4c20-8fe7-07c6f561b855-sys\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:36.383920 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.383873    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.383920 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.383899    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b24f6847-62c1-4f87-b320-80327be23ca9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:36.384021 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.383931    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65dfab40-fb6d-46d7-98f1-8243e194c176-host-slash\") pod \"iptables-alerter-mjlgj\" (UID: \"65dfab40-fb6d-46d7-98f1-8243e194c176\") " pod="openshift-network-operator/iptables-alerter-mjlgj"
Nov 02 01:25:36.384021 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.383943    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.384021 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.383943    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac453c3c-9209-4c20-8fe7-07c6f561b855-sys\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:36.384021 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.383963    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-systemd-units\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.384021 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.383973    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b24f6847-62c1-4f87-b320-80327be23ca9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:36.384021 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.383992    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-run-systemd\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.384021 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384008    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-sys\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.384210 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384027    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:36.384210 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384043    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b24f6847-62c1-4f87-b320-80327be23ca9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:36.384210 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.383993    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-systemd-units\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.384210 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384087    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-run-systemd\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.384210 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384103    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-sys\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.384210 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384113    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-sysctl-conf\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.384210 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384133    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-systemd\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.384210 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384150    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-lib-modules\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.384210 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384170    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-config-volume\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:36.384210 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384197    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:36.384210 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384204    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-systemd\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.384615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384225    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-var-lib-openvswitch\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.384615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384247    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-multus-conf-dir\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.384615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384258    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-lib-modules\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.384615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384272    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-var-lib-kubelet\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.384615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384313    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b49b523f-d706-4d6c-aac0-020b2e75c88a-audit-log\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:36.384615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384321    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-var-lib-openvswitch\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.384615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384329    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-multus-conf-dir\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.384615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384340    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:25:36.384615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384345    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-sysctl-conf\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.384615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384367    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-sys-fs\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:25:36.384615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384368    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-var-lib-kubelet\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.384615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384391    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:25:36.384615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384398    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-sys-fs\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:25:36.384615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384401    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:36.384615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384434    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6238ba5-f475-4e61-94d8-7d7bc13b2458-system-cni-dir\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:25:36.384615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384459    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-run-openvswitch\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.384615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384394    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b49b523f-d706-4d6c-aac0-020b2e75c88a-audit-log\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:36.384615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384489    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b49b523f-d706-4d6c-aac0-020b2e75c88a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:36.384615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384499    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6238ba5-f475-4e61-94d8-7d7bc13b2458-system-cni-dir\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:25:36.384615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384515    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-run-openvswitch\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.384615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384517    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdlxn\" (UniqueName: \"kubernetes.io/projected/b24f6847-62c1-4f87-b320-80327be23ca9-kube-api-access-bdlxn\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:36.384615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384548    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-kubelet\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.384615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384574    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-slash\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.384615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384608    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-kubelet\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.384615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384613    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-run\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384650    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-run\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384659    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-slash\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384662    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-run-multus-certs\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384691    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-var-lib-cni-multus\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384696    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-run-multus-certs\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384725    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-var-lib-cni-multus\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384730    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-run-netns\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384759    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-run-netns\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384767    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-cni-bin\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384792    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5cc4f303-fe62-44e7-ac7c-719defde064e-crio-socket\") pod \"insights-runtime-extractor-vrpsb\" (UID: \"5cc4f303-fe62-44e7-ac7c-719defde064e\") " pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384807    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-cnibin\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384823    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384861    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-modprobe-d\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384866    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5cc4f303-fe62-44e7-ac7c-719defde064e-crio-socket\") pod \"insights-runtime-extractor-vrpsb\" (UID: \"5cc4f303-fe62-44e7-ac7c-719defde064e\") " pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384883    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-cni-bin\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384888    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b0d87141-a1da-4d64-b01e-e66d8ba5e9ac-hosts-file\") pod \"node-resolver-bflth\" (UID: \"b0d87141-a1da-4d64-b01e-e66d8ba5e9ac\") " pod="openshift-dns/node-resolver-bflth"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384911    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6238ba5-f475-4e61-94d8-7d7bc13b2458-cnibin\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384926    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-sysconfig\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384958    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384883    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-cnibin\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384969    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-modprobe-d\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384984    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6238ba5-f475-4e61-94d8-7d7bc13b2458-cnibin\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.384997    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-var-lib-kubelet\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385007    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b0d87141-a1da-4d64-b01e-e66d8ba5e9ac-hosts-file\") pod \"node-resolver-bflth\" (UID: \"b0d87141-a1da-4d64-b01e-e66d8ba5e9ac\") " pod="openshift-dns/node-resolver-bflth"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385017    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-socket-dir\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385014    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-sysconfig\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385032    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-var-lib-kubelet\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385054    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6238ba5-f475-4e61-94d8-7d7bc13b2458-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385078    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-hostroot\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385087    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-socket-dir\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385096    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6238ba5-f475-4e61-94d8-7d7bc13b2458-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385096    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-grpc-tls\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385126    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-cni-netd\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385133    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-hostroot\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385144    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-host\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385170    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-system-cni-dir\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385171    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-cni-netd\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385177    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-host\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385195    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b24f6847-62c1-4f87-b320-80327be23ca9-config-out\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385225    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-system-cni-dir\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385226    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385262    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-run-ovn-kubernetes\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385306    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzlrg\" (UniqueName: \"kubernetes.io/projected/b49b523f-d706-4d6c-aac0-020b2e75c88a-kube-api-access-jzlrg\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385332    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b24f6847-62c1-4f87-b320-80327be23ca9-config-out\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385346    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-host-run-ovn-kubernetes\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385333    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385398    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-var-lib-cni-bin\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385425    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-web-config\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385454    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-registration-dir\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385475    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-var-lib-cni-bin\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385481    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-tls\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385514    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b49b523f-d706-4d6c-aac0-020b2e75c88a-secret-metrics-server-tls\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385519    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-registration-dir\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385542    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-run-netns\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385569    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-device-dir\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:25:36.385552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385594    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-run-netns\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385592    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385621    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-device-dir\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385693    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-run-k8s-cni-cncf-io\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385722    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-host-run-k8s-cni-cncf-io\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385721    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d352c5c1-44a0-4929-a43a-30c723958ce7-host\") pod \"node-ca-9lvcx\" (UID: \"d352c5c1-44a0-4929-a43a-30c723958ce7\") " pod="openshift-image-registry/node-ca-9lvcx"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385753    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d352c5c1-44a0-4929-a43a-30c723958ce7-host\") pod \"node-ca-9lvcx\" (UID: \"d352c5c1-44a0-4929-a43a-30c723958ce7\") " pod="openshift-image-registry/node-ca-9lvcx"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385762    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b24f6847-62c1-4f87-b320-80327be23ca9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385782    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-run-ovn\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385801    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-etc-openvswitch\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385816    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-multus-socket-dir-parent\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385826    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-run-ovn\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385832    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b49b523f-d706-4d6c-aac0-020b2e75c88a-client-ca-bundle\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385851    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-kubelet-config\") pod \"global-pull-secret-syncer-9kqsh\" (UID: \"26cf53e3-9cf3-4ad5-afe2-5c241f6749ad\") " pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385855    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-etc-openvswitch\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385868    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-wtmp\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385883    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-multus-cni-dir\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385891    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-multus-socket-dir-parent\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385902    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-etc-kubernetes\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385922    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-tmp\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385940    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-kubelet-config\") pod \"global-pull-secret-syncer-9kqsh\" (UID: \"26cf53e3-9cf3-4ad5-afe2-5c241f6749ad\") " pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385948    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-sysctl-d\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385956    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-etc-kubernetes\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385961    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-multus-cni-dir\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385951    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-wtmp\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.385986    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-metrics-client-ca\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.386007    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-tmp\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.386011    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp2pv\" (UniqueName: \"kubernetes.io/projected/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-kube-api-access-pp2pv\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.386052    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac453c3c-9209-4c20-8fe7-07c6f561b855-root\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.386084    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-node-log\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.386098    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac453c3c-9209-4c20-8fe7-07c6f561b855-root\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.386057    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-sysctl-d\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.386111    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-log-socket\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.386144    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-log-socket\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.386152    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b49b523f-d706-4d6c-aac0-020b2e75c88a-secret-metrics-server-client-certs\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.386119    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ef8451f5-e54e-42e3-aad7-16e13fb70d81-node-log\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.386188    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-dbus\") pod \"global-pull-secret-syncer-9kqsh\" (UID: \"26cf53e3-9cf3-4ad5-afe2-5c241f6749ad\") " pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.386218    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-etc-selinux\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.386248    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.386272    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-dbus\") pod \"global-pull-secret-syncer-9kqsh\" (UID: \"26cf53e3-9cf3-4ad5-afe2-5c241f6749ad\") " pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.386298    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b24f6847-62c1-4f87-b320-80327be23ca9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.386302    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7407afb5-5371-46c7-93ad-896d94a6f015-etc-selinux\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.386368    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-kubernetes\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.386390    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b49b523f-d706-4d6c-aac0-020b2e75c88a-metrics-server-audit-profiles\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.386456    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9256ae6-c79a-46f6-b6bd-8176b8907045-etc-kubernetes\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.386579    5639 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.386635    5639 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.386679    5639 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.386634    5639 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-53.us-west-1.compute.internal"
Nov 02 01:25:36.387381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.386997    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzgv6\" (UniqueName: \"kubernetes.io/projected/4c62a0f0-aefd-44fc-95c1-b2938317ee0d-kube-api-access-xzgv6\") pod \"multus-n5rzm\" (UID: \"4c62a0f0-aefd-44fc-95c1-b2938317ee0d\") " pod="openshift-multus/multus-n5rzm"
Nov 02 01:25:36.391483 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.391466    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwct4\" (UniqueName: \"kubernetes.io/projected/b6238ba5-f475-4e61-94d8-7d7bc13b2458-kube-api-access-zwct4\") pod \"multus-additional-cni-plugins-kmzfl\" (UID: \"b6238ba5-f475-4e61-94d8-7d7bc13b2458\") " pod="openshift-multus/multus-additional-cni-plugins-kmzfl"
Nov 02 01:25:36.419469 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.419396    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 01:25:36.419469 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.419450    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 01:25:36.419469 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.419450    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 01:25:36.419611 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.419496    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:25:36.420042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.419820    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 01:25:36.420042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.419842    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 01:25:36.420042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.419854    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 01:25:36.420042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.419882    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 01:25:36.420042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.419951    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 01:25:36.420042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.420042    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 01:25:36.420326 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.420077    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 01:25:36.420537 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.420387    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 02 01:25:36.420537 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.420399    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 02 01:25:36.420537 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.420413    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 02 01:25:36.420537 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.420444    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ef8451f5-e54e-42e3-aad7-16e13fb70d81-env-overrides\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.420537 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.420477    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 02 01:25:36.420537 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.420490    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 02 01:25:36.420537 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.420496    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 02 01:25:36.420537 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.420520    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 02 01:25:36.420537 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.420537    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 02 01:25:36.420983 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.420642    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gh69v\""
Nov 02 01:25:36.420983 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.420781    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 01:25:36.420983 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.420855    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 01:25:36.420983 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.420861    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wwr4r\""
Nov 02 01:25:36.420983 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.420893    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 01:25:36.420983 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.420913    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 01:25:36.420983 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.420924    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 01:25:36.420983 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.420935    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 01:25:36.420983 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.420939    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-tl75j\""
Nov 02 01:25:36.420983 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.420956    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 01:25:36.420983 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.420968    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 01:25:36.420983 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.420969    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 01:25:36.420983 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.420976    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 01:25:36.421528 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.421015    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 01:25:36.421528 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.421153    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 01:25:36.421528 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.421169    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 02 01:25:36.421528 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.421192    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 01:25:36.421528 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.421239    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 01:25:36.421528 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.421240    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 02 01:25:36.421528 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.421294    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/929a4d2e-02fa-4d5f-ace0-0ad3d41614b7-konnectivity-ca\") pod \"konnectivity-agent-vlg9k\" (UID: \"929a4d2e-02fa-4d5f-ace0-0ad3d41614b7\") " pod="kube-system/konnectivity-agent-vlg9k"
Nov 02 01:25:36.421528 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.421321    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 01:25:36.421528 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.421334    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 01:25:36.421528 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.421345    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 01:25:36.421528 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.421302    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 01:25:36.421528 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.421382    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 02 01:25:36.421528 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.421391    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 02 01:25:36.421528 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.421468    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 02 01:25:36.421528 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.421491    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ef8451f5-e54e-42e3-aad7-16e13fb70d81-ovnkube-config\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.421528 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.421507    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 02 01:25:36.422148 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.421560    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 02 01:25:36.422148 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.421572    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 02 01:25:36.422148 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.421602    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 02 01:25:36.422148 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.421613    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 02 01:25:36.422148 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.421700    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-82k56\""
Nov 02 01:25:36.422148 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.421743    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 01:25:36.422148 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.421747    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 01:25:36.422148 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.421908    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dck75\" (UniqueName: \"kubernetes.io/projected/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa-kube-api-access-dck75\") pod \"dns-default-mjwbd\" (UID: \"dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa\") " pod="openshift-dns/dns-default-mjwbd"
Nov 02 01:25:36.422148 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.421930    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrkz9\" (UniqueName: \"kubernetes.io/projected/d352c5c1-44a0-4929-a43a-30c723958ce7-kube-api-access-mrkz9\") pod \"node-ca-9lvcx\" (UID: \"d352c5c1-44a0-4929-a43a-30c723958ce7\") " pod="openshift-image-registry/node-ca-9lvcx"
Nov 02 01:25:36.422148 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.421983    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnlcv\" (UniqueName: \"kubernetes.io/projected/de6949bb-b875-4ca6-b0d3-19c07290bfae-kube-api-access-jnlcv\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:36.422148 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.422042    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5cc4f303-fe62-44e7-ac7c-719defde064e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vrpsb\" (UID: \"5cc4f303-fe62-44e7-ac7c-719defde064e\") " pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:36.422148 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.422137    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-tls\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:36.422664 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.422349    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de6949bb-b875-4ca6-b0d3-19c07290bfae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:36.422664 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.422407    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsgjl\" (UniqueName: \"kubernetes.io/projected/c9256ae6-c79a-46f6-b6bd-8176b8907045-kube-api-access-nsgjl\") pod \"tuned-stb4n\" (UID: \"c9256ae6-c79a-46f6-b6bd-8176b8907045\") " pod="openshift-cluster-node-tuning-operator/tuned-stb4n"
Nov 02 01:25:36.422664 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.422421    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa-config-volume\") pod \"dns-default-mjwbd\" (UID: \"dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa\") " pod="openshift-dns/dns-default-mjwbd"
Nov 02 01:25:36.422664 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.422535    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23f2f356-b80a-4e8c-a002-c09c006da8a1-cert\") pod \"ingress-canary-bncw7\" (UID: \"23f2f356-b80a-4e8c-a002-c09c006da8a1\") " pod="openshift-ingress-canary/ingress-canary-bncw7"
Nov 02 01:25:36.422664 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.422631    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d352c5c1-44a0-4929-a43a-30c723958ce7-serviceca\") pod \"node-ca-9lvcx\" (UID: \"d352c5c1-44a0-4929-a43a-30c723958ce7\") " pod="openshift-image-registry/node-ca-9lvcx"
Nov 02 01:25:36.422860 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.422845    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkznv\" (UniqueName: \"kubernetes.io/projected/5cc4f303-fe62-44e7-ac7c-719defde064e-kube-api-access-mkznv\") pod \"insights-runtime-extractor-vrpsb\" (UID: \"5cc4f303-fe62-44e7-ac7c-719defde064e\") " pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:36.425233 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.425193    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-config-volume\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:36.425233 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.425203    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:36.425233 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.425200    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:36.425393 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.425249    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b49b523f-d706-4d6c-aac0-020b2e75c88a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:36.425393 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.425257    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:36.425393 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.425298    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:36.425393 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.425250    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:36.426044 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.426023    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:36.426110 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.426044    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:36.426110 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.426072    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-web-config\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:36.426110 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.426081    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:36.426110 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.426077    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b49b523f-d706-4d6c-aac0-020b2e75c88a-secret-metrics-server-tls\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:36.426556 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.426537    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fswd\" (UniqueName: \"kubernetes.io/projected/b0d87141-a1da-4d64-b01e-e66d8ba5e9ac-kube-api-access-9fswd\") pod \"node-resolver-bflth\" (UID: \"b0d87141-a1da-4d64-b01e-e66d8ba5e9ac\") " pod="openshift-dns/node-resolver-bflth"
Nov 02 01:25:36.426652 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.426629    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2n6b\" (UniqueName: \"kubernetes.io/projected/bc6acc82-57d4-4906-9c57-67d717e89ba3-kube-api-access-c2n6b\") pod \"openshift-state-metrics-7bd7489957-cj277\" (UID: \"bc6acc82-57d4-4906-9c57-67d717e89ba3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277"
Nov 02 01:25:36.426713 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.426681    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b24f6847-62c1-4f87-b320-80327be23ca9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:36.426866 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.426845    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:36.426866 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.426848    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b49b523f-d706-4d6c-aac0-020b2e75c88a-metrics-server-audit-profiles\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:36.426994 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.426847    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b49b523f-d706-4d6c-aac0-020b2e75c88a-secret-metrics-server-client-certs\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:36.426994 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.426953    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft2nz\" (UniqueName: \"kubernetes.io/projected/ac453c3c-9209-4c20-8fe7-07c6f561b855-kube-api-access-ft2nz\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:36.427075 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.427060    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p47d\" (UniqueName: \"kubernetes.io/projected/7407afb5-5371-46c7-93ad-896d94a6f015-kube-api-access-9p47d\") pod \"aws-ebs-csi-driver-node-tt978\" (UID: \"7407afb5-5371-46c7-93ad-896d94a6f015\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tt978"
Nov 02 01:25:36.427222 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.427208    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9w59\" (UniqueName: \"kubernetes.io/projected/4240f2d9-dd99-4ca9-8447-7706cebcfa08-kube-api-access-p9w59\") pod \"network-check-target-w4jf8\" (UID: \"4240f2d9-dd99-4ca9-8447-7706cebcfa08\") " pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:25:36.429069 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.429052    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:36.430160 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.430144    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ac453c3c-9209-4c20-8fe7-07c6f561b855-node-exporter-accelerators-collector-config\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:36.430160 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.430153    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eb605c8-f995-49ae-a1b8-06aa55160b13-metrics-certs\") pod \"network-metrics-daemon-rfwwn\" (UID: \"9eb605c8-f995-49ae-a1b8-06aa55160b13\") " pod="openshift-multus/network-metrics-daemon-rfwwn"
Nov 02 01:25:36.430248 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.430159    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ef8451f5-e54e-42e3-aad7-16e13fb70d81-ovn-node-metrics-cert\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.430824 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.430804    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc6acc82-57d4-4906-9c57-67d717e89ba3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-cj277\" (UID: \"bc6acc82-57d4-4906-9c57-67d717e89ba3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277"
Nov 02 01:25:36.430883 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.430812    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa-metrics-tls\") pod \"dns-default-mjwbd\" (UID: \"dd59a74b-47c5-4e19-a2e4-09a8f1ec29aa\") " pod="openshift-dns/dns-default-mjwbd"
Nov 02 01:25:36.431235 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.431221    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jd7v\" (UniqueName: \"kubernetes.io/projected/23f2f356-b80a-4e8c-a002-c09c006da8a1-kube-api-access-6jd7v\") pod \"ingress-canary-bncw7\" (UID: \"23f2f356-b80a-4e8c-a002-c09c006da8a1\") " pod="openshift-ingress-canary/ingress-canary-bncw7"
Nov 02 01:25:36.441012 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.440991    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pw4q8\""
Nov 02 01:25:36.441099 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.441036    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 01:25:36.441245 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.441230    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 02 01:25:36.441335 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.441257    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wfbzc\""
Nov 02 01:25:36.441335 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.441293    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2dvkb\""
Nov 02 01:25:36.442706 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.442690    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj9pc\" (UniqueName: \"kubernetes.io/projected/ef8451f5-e54e-42e3-aad7-16e13fb70d81-kube-api-access-gj9pc\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.444829 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.444812    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b24f6847-62c1-4f87-b320-80327be23ca9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:36.468251 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.468228    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6qqtd\""
Nov 02 01:25:36.469178 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.469162    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q5j6l\""
Nov 02 01:25:36.469733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.469501    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 01:25:36.469733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.469503    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-tlxc4\""
Nov 02 01:25:36.469733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.469531    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-evgl73d9t320t\""
Nov 02 01:25:36.469733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.469530    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-txd4k\""
Nov 02 01:25:36.469733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.469548    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jpwmt\""
Nov 02 01:25:36.469733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.469557    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 01:25:36.469733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.469509    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l67lg\""
Nov 02 01:25:36.469733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.469569    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fr4ur72tdnr05\""
Nov 02 01:25:36.469733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.469564    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 02 01:25:36.469733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.469533    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 02 01:25:36.469733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.469603    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 01:25:36.469733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.469609    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 01:25:36.469733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.469615    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 01:25:36.469733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.469610    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4dgbb\""
Nov 02 01:25:36.469733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.469658    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-2vxps\""
Nov 02 01:25:36.469733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.469684    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 02 01:25:36.469733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.469692    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p5zpq\""
Nov 02 01:25:36.469733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.469666    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qtccq\""
Nov 02 01:25:36.470588 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.469821    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de6949bb-b875-4ca6-b0d3-19c07290bfae-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:36.470588 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.469867    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jpr5g\""
Nov 02 01:25:36.470588 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.469925    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 02 01:25:36.470588 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.470043    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/929a4d2e-02fa-4d5f-ace0-0ad3d41614b7-agent-certs\") pod \"konnectivity-agent-vlg9k\" (UID: \"929a4d2e-02fa-4d5f-ace0-0ad3d41614b7\") " pod="kube-system/konnectivity-agent-vlg9k"
Nov 02 01:25:36.470588 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.470270    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzlrg\" (UniqueName: \"kubernetes.io/projected/b49b523f-d706-4d6c-aac0-020b2e75c88a-kube-api-access-jzlrg\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:36.470588 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.470420    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdlxn\" (UniqueName: \"kubernetes.io/projected/b24f6847-62c1-4f87-b320-80327be23ca9-kube-api-access-bdlxn\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:36.470588 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.470473    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp2pv\" (UniqueName: \"kubernetes.io/projected/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-kube-api-access-pp2pv\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:36.470811 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.470744    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26cf53e3-9cf3-4ad5-afe2-5c241f6749ad-original-pull-secret\") pod \"global-pull-secret-syncer-9kqsh\" (UID: \"26cf53e3-9cf3-4ad5-afe2-5c241f6749ad\") " pod="kube-system/global-pull-secret-syncer-9kqsh"
Nov 02 01:25:36.472223 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.472203    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/de6949bb-b875-4ca6-b0d3-19c07290bfae-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:36.472313 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.472301    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ef8451f5-e54e-42e3-aad7-16e13fb70d81-ovnkube-script-lib\") pod \"ovnkube-node-4vjq9\" (UID: \"ef8451f5-e54e-42e3-aad7-16e13fb70d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:36.472349 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.472309    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc6acc82-57d4-4906-9c57-67d717e89ba3-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-cj277\" (UID: \"bc6acc82-57d4-4906-9c57-67d717e89ba3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277"
Nov 02 01:25:36.473424 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.473410    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac453c3c-9209-4c20-8fe7-07c6f561b855-metrics-client-ca\") pod \"node-exporter-s59gt\" (UID: \"ac453c3c-9209-4c20-8fe7-07c6f561b855\") " pod="openshift-monitoring/node-exporter-s59gt"
Nov 02 01:25:36.473475 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.473430    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc6acc82-57d4-4906-9c57-67d717e89ba3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-cj277\" (UID: \"bc6acc82-57d4-4906-9c57-67d717e89ba3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cj277"
Nov 02 01:25:36.473475 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.473437    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5cc4f303-fe62-44e7-ac7c-719defde064e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vrpsb\" (UID: \"5cc4f303-fe62-44e7-ac7c-719defde064e\") " pod="openshift-insights/insights-runtime-extractor-vrpsb"
Nov 02 01:25:36.475609 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.475590    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-grpc-tls\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:36.475788 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.475773    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-secret-thanos-querier-tls\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:36.476720 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.476701    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d-metrics-client-ca\") pod \"thanos-querier-c68bc9867-x88cx\" (UID: \"f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d\") " pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:36.476801 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.476723    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b24f6847-62c1-4f87-b320-80327be23ca9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:25:36.476836 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.476808    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b49b523f-d706-4d6c-aac0-020b2e75c88a-client-ca-bundle\") pod \"metrics-server-5bfddd89bf-5vgk2\" (UID: \"b49b523f-d706-4d6c-aac0-020b2e75c88a\") " pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:36.480010 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.479995    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/de6949bb-b875-4ca6-b0d3-19c07290bfae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-g7f8f\" (UID: \"de6949bb-b875-4ca6-b0d3-19c07290bfae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g7f8f"
Nov 02 01:25:36.675921 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.675841    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-c68bc9867-x88cx"
Nov 02 01:25:36.790400 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.790360    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-serving-cert\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:36.790400 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.790402    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-oauth-config\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:36.790587 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.790431    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-console-config\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:36.790587 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.790480    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-service-ca\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:36.790587 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.790510    5639 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 02 01:25:36.790587 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.790542    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-oauth-serving-cert\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:36.790587 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.790569    5639 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 02 01:25:36.790587 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.790577    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-serving-cert podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:37.790563149 +0000 UTC m=+2.694245240 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-serving-cert") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"console-serving-cert" not registered
Nov 02 01:25:36.790587 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.790571    5639 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 02 01:25:36.790776 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.790602    5639 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 02 01:25:36.790776 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.790609    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-service-ca podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:37.790601409 +0000 UTC m=+2.694283502 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-service-ca") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"service-ca" not registered
Nov 02 01:25:36.790776 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.790510    5639 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 02 01:25:36.790776 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.790631    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-console-config podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:37.790614138 +0000 UTC m=+2.694296254 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-console-config") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"console-config" not registered
Nov 02 01:25:36.790776 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.790647    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-oauth-config podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:37.790638771 +0000 UTC m=+2.694320864 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-oauth-config") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"console-oauth-config" not registered
Nov 02 01:25:36.790776 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.790722    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-oauth-serving-cert podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:37.790709077 +0000 UTC m=+2.694391172 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-oauth-serving-cert") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 02 01:25:36.891004 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:36.890969    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfdq8\" (UniqueName: \"kubernetes.io/projected/887b1b53-10f5-4392-89a5-f90a431e60ea-kube-api-access-qfdq8\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:36.891179 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.891127    5639 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 02 01:25:36.891179 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.891145    5639 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 02 01:25:36.891179 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.891156    5639 projected.go:196] Error preparing data for projected volume kube-api-access-qfdq8 for pod openshift-console/console-86d9455949-gsxx6: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 02 01:25:36.891267 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:36.891220    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/887b1b53-10f5-4392-89a5-f90a431e60ea-kube-api-access-qfdq8 podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:37.891203093 +0000 UTC m=+2.794885199 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qfdq8" (UniqueName: "kubernetes.io/projected/887b1b53-10f5-4392-89a5-f90a431e60ea-kube-api-access-qfdq8") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 02 01:25:37.788395 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:37.788354    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mjwbd"
Nov 02 01:25:37.800080 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:37.800053    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-oauth-serving-cert\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:37.800192 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:37.800088    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-serving-cert\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:37.800192 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:37.800111    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-oauth-config\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:37.800192 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:37.800134    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-console-config\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:37.800192 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:37.800167    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-service-ca\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:37.800407 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:37.800252    5639 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 02 01:25:37.800407 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:37.800301    5639 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 02 01:25:37.800407 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:37.800340    5639 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 02 01:25:37.800407 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:37.800383    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-console-config podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:39.800353626 +0000 UTC m=+4.704035732 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-console-config") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"console-config" not registered
Nov 02 01:25:37.800586 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:37.800412    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-oauth-serving-cert podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:39.800398026 +0000 UTC m=+4.704080133 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-oauth-serving-cert") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 02 01:25:37.800586 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:37.800432    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-serving-cert podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:39.800420301 +0000 UTC m=+4.704102398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-serving-cert") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"console-serving-cert" not registered
Nov 02 01:25:37.800586 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:37.800443    5639 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 02 01:25:37.800586 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:37.800468    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-oauth-config podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:39.800458428 +0000 UTC m=+4.704140523 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-oauth-config") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"console-oauth-config" not registered
Nov 02 01:25:37.800586 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:37.800481    5639 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 02 01:25:37.800586 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:37.800524    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-service-ca podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:39.800514524 +0000 UTC m=+4.704196636 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-service-ca") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"service-ca" not registered
Nov 02 01:25:37.901099 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:37.901056    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfdq8\" (UniqueName: \"kubernetes.io/projected/887b1b53-10f5-4392-89a5-f90a431e60ea-kube-api-access-qfdq8\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:37.901337 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:37.901241    5639 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 02 01:25:37.901337 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:37.901260    5639 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 02 01:25:37.901337 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:37.901270    5639 projected.go:196] Error preparing data for projected volume kube-api-access-qfdq8 for pod openshift-console/console-86d9455949-gsxx6: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 02 01:25:37.901478 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:37.901347    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/887b1b53-10f5-4392-89a5-f90a431e60ea-kube-api-access-qfdq8 podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:39.901333762 +0000 UTC m=+4.805015854 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qfdq8" (UniqueName: "kubernetes.io/projected/887b1b53-10f5-4392-89a5-f90a431e60ea-kube-api-access-qfdq8") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 02 01:25:39.818806 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:39.818763    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-console-config\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:39.819361 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:39.818839    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-service-ca\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:39.819361 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:39.818893    5639 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 02 01:25:39.819361 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:39.818908    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-oauth-serving-cert\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:39.819361 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:39.818940    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-serving-cert\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:39.819361 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:39.818955    5639 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 02 01:25:39.819361 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:39.818959    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-console-config podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:43.818943461 +0000 UTC m=+8.722625553 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-console-config") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"console-config" not registered
Nov 02 01:25:39.819361 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:39.819028    5639 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 02 01:25:39.819361 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:39.819032    5639 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 02 01:25:39.819361 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:39.819039    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-service-ca podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:43.819020909 +0000 UTC m=+8.722703008 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-service-ca") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"service-ca" not registered
Nov 02 01:25:39.819361 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:39.819106    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-oauth-serving-cert podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:43.819089756 +0000 UTC m=+8.722771860 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-oauth-serving-cert") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 02 01:25:39.819361 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:39.819138    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-serving-cert podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:43.819128029 +0000 UTC m=+8.722810127 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-serving-cert") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"console-serving-cert" not registered
Nov 02 01:25:39.819361 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:39.819177    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-oauth-config\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:39.819361 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:39.819237    5639 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 02 01:25:39.819361 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:39.819270    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-oauth-config podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:43.819262048 +0000 UTC m=+8.722944170 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-oauth-config") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"console-oauth-config" not registered
Nov 02 01:25:39.920506 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:39.920468    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfdq8\" (UniqueName: \"kubernetes.io/projected/887b1b53-10f5-4392-89a5-f90a431e60ea-kube-api-access-qfdq8\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:39.920656 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:39.920611    5639 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 02 01:25:39.920656 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:39.920627    5639 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 02 01:25:39.920656 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:39.920635    5639 projected.go:196] Error preparing data for projected volume kube-api-access-qfdq8 for pod openshift-console/console-86d9455949-gsxx6: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 02 01:25:39.920759 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:39.920686    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/887b1b53-10f5-4392-89a5-f90a431e60ea-kube-api-access-qfdq8 podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:43.920674127 +0000 UTC m=+8.824356232 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qfdq8" (UniqueName: "kubernetes.io/projected/887b1b53-10f5-4392-89a5-f90a431e60ea-kube-api-access-qfdq8") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 02 01:25:40.096434 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:40.096404    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:25:40.811404 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:40.811377    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:43.534934 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:43.534906    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5bfddd89bf-5vgk2"
Nov 02 01:25:43.860342 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:43.860258    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-oauth-serving-cert\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:43.860499 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:43.860358    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-serving-cert\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:43.860499 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:43.860387    5639 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 02 01:25:43.860499 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:43.860457    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-oauth-serving-cert podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:51.860439674 +0000 UTC m=+16.764121785 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-oauth-serving-cert") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 02 01:25:43.860499 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:43.860478    5639 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 02 01:25:43.860678 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:43.860392    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-oauth-config\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:43.860678 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:43.860495    5639 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 02 01:25:43.860678 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:43.860525    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-oauth-config podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:51.86051478 +0000 UTC m=+16.764196892 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-oauth-config") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"console-oauth-config" not registered
Nov 02 01:25:43.860678 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:43.860569    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-console-config\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:43.860678 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:43.860584    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-serving-cert podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:51.860562468 +0000 UTC m=+16.764244574 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-serving-cert") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"console-serving-cert" not registered
Nov 02 01:25:43.860678 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:43.860608    5639 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 02 01:25:43.860678 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:43.860652    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-console-config podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:51.860642396 +0000 UTC m=+16.764324491 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-console-config") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"console-config" not registered
Nov 02 01:25:43.860875 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:43.860680    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-service-ca\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:43.860875 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:43.860765    5639 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 02 01:25:43.860875 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:43.860810    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-service-ca podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:51.860802436 +0000 UTC m=+16.764484532 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-service-ca") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"service-ca" not registered
Nov 02 01:25:43.962192 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:43.962149    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfdq8\" (UniqueName: \"kubernetes.io/projected/887b1b53-10f5-4392-89a5-f90a431e60ea-kube-api-access-qfdq8\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:43.962386 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:43.962332    5639 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 02 01:25:43.962386 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:43.962357    5639 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 02 01:25:43.962386 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:43.962368    5639 projected.go:196] Error preparing data for projected volume kube-api-access-qfdq8 for pod openshift-console/console-86d9455949-gsxx6: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 02 01:25:43.962485 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:43.962419    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/887b1b53-10f5-4392-89a5-f90a431e60ea-kube-api-access-qfdq8 podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:25:51.962406077 +0000 UTC m=+16.866088173 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qfdq8" (UniqueName: "kubernetes.io/projected/887b1b53-10f5-4392-89a5-f90a431e60ea-kube-api-access-qfdq8") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 02 01:25:46.482214 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:46.482122    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:25:46.482693 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:46.482406    5639 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 01:25:46.812029 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:46.811951    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vlg9k"
Nov 02 01:25:51.937947 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:51.937902    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-oauth-serving-cert\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:51.937947 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:51.937951    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-serving-cert\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:51.938684 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:51.937988    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-oauth-config\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:51.938684 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:51.938019    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-console-config\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:51.938684 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:51.938025    5639 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 02 01:25:51.938684 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:51.938082    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-service-ca\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:51.938684 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:51.938113    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-oauth-serving-cert podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:26:07.938093139 +0000 UTC m=+32.841775251 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-oauth-serving-cert") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 02 01:25:51.938684 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:51.938138    5639 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 02 01:25:51.938684 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:51.938184    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-oauth-config podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:26:07.938172211 +0000 UTC m=+32.841854317 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-oauth-config") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"console-oauth-config" not registered
Nov 02 01:25:51.938684 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:51.938138    5639 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 02 01:25:51.938684 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:51.938199    5639 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 02 01:25:51.938684 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:51.938138    5639 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 02 01:25:51.938684 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:51.938219    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-service-ca podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:26:07.938212862 +0000 UTC m=+32.841894957 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-service-ca") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"service-ca" not registered
Nov 02 01:25:51.938684 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:51.938314    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-console-config podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:26:07.938273903 +0000 UTC m=+32.841955997 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-console-config") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"console-config" not registered
Nov 02 01:25:51.938684 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:51.938329    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-serving-cert podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:26:07.93832328 +0000 UTC m=+32.842005372 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-serving-cert") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : object "openshift-console"/"console-serving-cert" not registered
Nov 02 01:25:52.039368 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:52.039316    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfdq8\" (UniqueName: \"kubernetes.io/projected/887b1b53-10f5-4392-89a5-f90a431e60ea-kube-api-access-qfdq8\") pod \"console-86d9455949-gsxx6\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") " pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:25:52.039549 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:52.039484    5639 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 02 01:25:52.039549 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:52.039505    5639 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 02 01:25:52.039549 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:52.039516    5639 projected.go:196] Error preparing data for projected volume kube-api-access-qfdq8 for pod openshift-console/console-86d9455949-gsxx6: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 02 01:25:52.039703 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:52.039574    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/887b1b53-10f5-4392-89a5-f90a431e60ea-kube-api-access-qfdq8 podName:887b1b53-10f5-4392-89a5-f90a431e60ea nodeName:}" failed. No retries permitted until 2026-11-02 01:26:08.039559635 +0000 UTC m=+32.943241756 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qfdq8" (UniqueName: "kubernetes.io/projected/887b1b53-10f5-4392-89a5-f90a431e60ea-kube-api-access-qfdq8") pod "console-86d9455949-gsxx6" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 02 01:25:57.420000 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:25:57.419965    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:25:58.792337 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:25:58.792301    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vjq9"
Nov 02 01:26:01.295901 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:01.295853    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-86d9455949-gsxx6" podUID="887b1b53-10f5-4392-89a5-f90a431e60ea" containerName="console" containerID="cri-o://7afacb677ed1ee9f59cc7dd7a4e93676a0a6c45d113fd57a4a483efa568eb74b" gracePeriod=15
Nov 02 01:26:01.538696 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:01.538670    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:26:01.627171 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:01.627136    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-service-ca\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-service-ca\") pod \"887b1b53-10f5-4392-89a5-f90a431e60ea\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") "
Nov 02 01:26:01.627391 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:01.627182    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/887b1b53-10f5-4392-89a5-f90a431e60ea-kube-api-access-qfdq8\" (UniqueName: \"kubernetes.io/projected/887b1b53-10f5-4392-89a5-f90a431e60ea-kube-api-access-qfdq8\") pod \"887b1b53-10f5-4392-89a5-f90a431e60ea\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") "
Nov 02 01:26:01.627391 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:01.627221    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-serving-cert\") pod \"887b1b53-10f5-4392-89a5-f90a431e60ea\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") "
Nov 02 01:26:01.627514 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:01.627411    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-oauth-serving-cert\") pod \"887b1b53-10f5-4392-89a5-f90a431e60ea\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") "
Nov 02 01:26:01.627514 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:01.627445    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-console-config\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-console-config\") pod \"887b1b53-10f5-4392-89a5-f90a431e60ea\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") "
Nov 02 01:26:01.627628 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:01.627566    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-service-ca" pod "887b1b53-10f5-4392-89a5-f90a431e60ea" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:26:01.627628 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:01.627609    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-oauth-config\") pod \"887b1b53-10f5-4392-89a5-f90a431e60ea\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") "
Nov 02 01:26:01.627731 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:01.627687    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887b1b53-10f5-4392-89a5-f90a431e60ea-tmp\" (UniqueName: \"kubernetes.io/empty-dir/887b1b53-10f5-4392-89a5-f90a431e60ea-tmp\") pod \"887b1b53-10f5-4392-89a5-f90a431e60ea\" (UID: \"887b1b53-10f5-4392-89a5-f90a431e60ea\") "
Nov 02 01:26:01.627841 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:01.627796    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-oauth-serving-cert" pod "887b1b53-10f5-4392-89a5-f90a431e60ea" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:26:01.627841 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:01.627802    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-console-config" pod "887b1b53-10f5-4392-89a5-f90a431e60ea" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:26:01.628123 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:01.627967    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887b1b53-10f5-4392-89a5-f90a431e60ea-tmp" pod "887b1b53-10f5-4392-89a5-f90a431e60ea" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:26:01.628123 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:01.628038    5639 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-oauth-serving-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:01.628123 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:01.628056    5639 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-console-config\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:01.628123 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:01.628069    5639 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/887b1b53-10f5-4392-89a5-f90a431e60ea-service-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:01.629675 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:01.629655    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/887b1b53-10f5-4392-89a5-f90a431e60ea-kube-api-access-qfdq8" pod "887b1b53-10f5-4392-89a5-f90a431e60ea" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea"). InnerVolumeSpecName "kube-api-access-qfdq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:26:01.630073 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:01.630056    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-serving-cert" pod "887b1b53-10f5-4392-89a5-f90a431e60ea" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:26:01.630204 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:01.630135    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-oauth-config" pod "887b1b53-10f5-4392-89a5-f90a431e60ea" (UID: "887b1b53-10f5-4392-89a5-f90a431e60ea"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:26:01.728867 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:01.728834    5639 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-oauth-config\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:01.728867 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:01.728862    5639 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/887b1b53-10f5-4392-89a5-f90a431e60ea-tmp\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:01.728867 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:01.728872    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfdq8\" (UniqueName: \"kubernetes.io/projected/887b1b53-10f5-4392-89a5-f90a431e60ea-kube-api-access-qfdq8\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:01.729098 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:01.728881    5639 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/887b1b53-10f5-4392-89a5-f90a431e60ea-console-serving-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:02.368909 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:02.368874    5639 generic.go:356] "Generic (PLEG): container finished" podID="887b1b53-10f5-4392-89a5-f90a431e60ea" containerID="7afacb677ed1ee9f59cc7dd7a4e93676a0a6c45d113fd57a4a483efa568eb74b" exitCode=0
Nov 02 01:26:02.369454 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:02.368929    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86d9455949-gsxx6"
Nov 02 01:26:02.369454 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:02.368959    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86d9455949-gsxx6" event={"ID":"887b1b53-10f5-4392-89a5-f90a431e60ea","Type":"ContainerDied","Data":"7afacb677ed1ee9f59cc7dd7a4e93676a0a6c45d113fd57a4a483efa568eb74b"}
Nov 02 01:26:02.369454 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:02.369017    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86d9455949-gsxx6" event={"ID":"887b1b53-10f5-4392-89a5-f90a431e60ea","Type":"ContainerDied","Data":"94d9247bfa65f5f7793065e3f27f3ecfd7a081eaca2b2e6276d4acf815e12d9b"}
Nov 02 01:26:02.369454 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:02.369037    5639 scope.go:122] "RemoveContainer" containerID="7afacb677ed1ee9f59cc7dd7a4e93676a0a6c45d113fd57a4a483efa568eb74b"
Nov 02 01:26:02.379172 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:02.379142    5639 scope.go:122] "RemoveContainer" containerID="7afacb677ed1ee9f59cc7dd7a4e93676a0a6c45d113fd57a4a483efa568eb74b"
Nov 02 01:26:02.379488 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:26:02.379470    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7afacb677ed1ee9f59cc7dd7a4e93676a0a6c45d113fd57a4a483efa568eb74b\": container with ID starting with 7afacb677ed1ee9f59cc7dd7a4e93676a0a6c45d113fd57a4a483efa568eb74b not found: ID does not exist" containerID="7afacb677ed1ee9f59cc7dd7a4e93676a0a6c45d113fd57a4a483efa568eb74b"
Nov 02 01:26:02.379562 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:02.379497    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7afacb677ed1ee9f59cc7dd7a4e93676a0a6c45d113fd57a4a483efa568eb74b"} err="failed to get container status \"7afacb677ed1ee9f59cc7dd7a4e93676a0a6c45d113fd57a4a483efa568eb74b\": rpc error: code = NotFound desc = could not find container \"7afacb677ed1ee9f59cc7dd7a4e93676a0a6c45d113fd57a4a483efa568eb74b\": container with ID starting with 7afacb677ed1ee9f59cc7dd7a4e93676a0a6c45d113fd57a4a483efa568eb74b not found: ID does not exist"
Nov 02 01:26:02.532598 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:02.532565    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86d9455949-gsxx6"]
Nov 02 01:26:02.582215 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:02.582183    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86d9455949-gsxx6"]
Nov 02 01:26:03.215585 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:03.215555    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="887b1b53-10f5-4392-89a5-f90a431e60ea" path="/var/lib/kubelet/pods/887b1b53-10f5-4392-89a5-f90a431e60ea/volumes"
Nov 02 01:26:10.099725 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:10.099694    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-w4jf8"
Nov 02 01:26:19.246257 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:19.246225    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 01:26:19.246685 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:19.246523    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="alertmanager" containerID="cri-o://bf7dd96fc431d213b1729dd5e71942568d39bfdedc3c918b18b3bec8ed1de31f" gracePeriod=120
Nov 02 01:26:19.246685 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:19.246592    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="kube-rbac-proxy-metric" containerID="cri-o://2154e062960bb97fa61add68105bf163c102ca16cb7ff631642808e15955dbb8" gracePeriod=120
Nov 02 01:26:19.246685 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:19.246637    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="kube-rbac-proxy-web" containerID="cri-o://0f4531dc393e10959694446647655bccd77e1e3b837bb5ad6d0764ec41518de3" gracePeriod=120
Nov 02 01:26:19.246856 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:19.246665    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="config-reloader" containerID="cri-o://fef375051d2276b6faa3bffc7ee304a8a508397fc33754e4ef3290a121fe0139" gracePeriod=120
Nov 02 01:26:19.246856 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:19.246713    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="kube-rbac-proxy" containerID="cri-o://e4bd992cf245d9545be341c555b4e9faaf95c09115677dc2d433a11c19f1b0d5" gracePeriod=120
Nov 02 01:26:19.247750 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:19.246640    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="prom-label-proxy" containerID="cri-o://9009b04852505c6f7aecebfd17f4f7b23bc5ee793080eded82aa2c0a36c3da3f" gracePeriod=120
Nov 02 01:26:19.428422 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:19.428388    5639 generic.go:356] "Generic (PLEG): container finished" podID="b24f6847-62c1-4f87-b320-80327be23ca9" containerID="9009b04852505c6f7aecebfd17f4f7b23bc5ee793080eded82aa2c0a36c3da3f" exitCode=0
Nov 02 01:26:19.428422 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:19.428413    5639 generic.go:356] "Generic (PLEG): container finished" podID="b24f6847-62c1-4f87-b320-80327be23ca9" containerID="e4bd992cf245d9545be341c555b4e9faaf95c09115677dc2d433a11c19f1b0d5" exitCode=0
Nov 02 01:26:19.428422 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:19.428419    5639 generic.go:356] "Generic (PLEG): container finished" podID="b24f6847-62c1-4f87-b320-80327be23ca9" containerID="fef375051d2276b6faa3bffc7ee304a8a508397fc33754e4ef3290a121fe0139" exitCode=0
Nov 02 01:26:19.428422 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:19.428425    5639 generic.go:356] "Generic (PLEG): container finished" podID="b24f6847-62c1-4f87-b320-80327be23ca9" containerID="bf7dd96fc431d213b1729dd5e71942568d39bfdedc3c918b18b3bec8ed1de31f" exitCode=0
Nov 02 01:26:19.428666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:19.428447    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b24f6847-62c1-4f87-b320-80327be23ca9","Type":"ContainerDied","Data":"9009b04852505c6f7aecebfd17f4f7b23bc5ee793080eded82aa2c0a36c3da3f"}
Nov 02 01:26:19.428666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:19.428499    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b24f6847-62c1-4f87-b320-80327be23ca9","Type":"ContainerDied","Data":"e4bd992cf245d9545be341c555b4e9faaf95c09115677dc2d433a11c19f1b0d5"}
Nov 02 01:26:19.428666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:19.428525    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b24f6847-62c1-4f87-b320-80327be23ca9","Type":"ContainerDied","Data":"fef375051d2276b6faa3bffc7ee304a8a508397fc33754e4ef3290a121fe0139"}
Nov 02 01:26:19.428666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:19.428548    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b24f6847-62c1-4f87-b320-80327be23ca9","Type":"ContainerDied","Data":"bf7dd96fc431d213b1729dd5e71942568d39bfdedc3c918b18b3bec8ed1de31f"}
Nov 02 01:26:20.437102 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.437066    5639 generic.go:356] "Generic (PLEG): container finished" podID="b24f6847-62c1-4f87-b320-80327be23ca9" containerID="2154e062960bb97fa61add68105bf163c102ca16cb7ff631642808e15955dbb8" exitCode=0
Nov 02 01:26:20.437102 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.437102    5639 generic.go:356] "Generic (PLEG): container finished" podID="b24f6847-62c1-4f87-b320-80327be23ca9" containerID="0f4531dc393e10959694446647655bccd77e1e3b837bb5ad6d0764ec41518de3" exitCode=0
Nov 02 01:26:20.437535 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.437136    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b24f6847-62c1-4f87-b320-80327be23ca9","Type":"ContainerDied","Data":"2154e062960bb97fa61add68105bf163c102ca16cb7ff631642808e15955dbb8"}
Nov 02 01:26:20.437535 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.437220    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b24f6847-62c1-4f87-b320-80327be23ca9","Type":"ContainerDied","Data":"0f4531dc393e10959694446647655bccd77e1e3b837bb5ad6d0764ec41518de3"}
Nov 02 01:26:20.492521 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.492492    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:20.567419 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.567337    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24f6847-62c1-4f87-b320-80327be23ca9-config-out\" (UniqueName: \"kubernetes.io/empty-dir/b24f6847-62c1-4f87-b320-80327be23ca9-config-out\") pod \"b24f6847-62c1-4f87-b320-80327be23ca9\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") "
Nov 02 01:26:20.567419 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.567375    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b24f6847-62c1-4f87-b320-80327be23ca9-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b24f6847-62c1-4f87-b320-80327be23ca9-metrics-client-ca\") pod \"b24f6847-62c1-4f87-b320-80327be23ca9\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") "
Nov 02 01:26:20.567419 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.567402    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-cluster-tls-config\") pod \"b24f6847-62c1-4f87-b320-80327be23ca9\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") "
Nov 02 01:26:20.567697 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.567423    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b24f6847-62c1-4f87-b320-80327be23ca9-kube-api-access-bdlxn\" (UniqueName: \"kubernetes.io/projected/b24f6847-62c1-4f87-b320-80327be23ca9-kube-api-access-bdlxn\") pod \"b24f6847-62c1-4f87-b320-80327be23ca9\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") "
Nov 02 01:26:20.567697 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.567461    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24f6847-62c1-4f87-b320-80327be23ca9-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b24f6847-62c1-4f87-b320-80327be23ca9-alertmanager-main-db\") pod \"b24f6847-62c1-4f87-b320-80327be23ca9\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") "
Nov 02 01:26:20.567697 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.567484    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b24f6847-62c1-4f87-b320-80327be23ca9-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b24f6847-62c1-4f87-b320-80327be23ca9-alertmanager-trusted-ca-bundle\") pod \"b24f6847-62c1-4f87-b320-80327be23ca9\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") "
Nov 02 01:26:20.567697 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.567508    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy-web\") pod \"b24f6847-62c1-4f87-b320-80327be23ca9\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") "
Nov 02 01:26:20.567697 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.567536    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-config-volume\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-config-volume\") pod \"b24f6847-62c1-4f87-b320-80327be23ca9\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") "
Nov 02 01:26:20.567697 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.567562    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy\") pod \"b24f6847-62c1-4f87-b320-80327be23ca9\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") "
Nov 02 01:26:20.567697 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.567620    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b24f6847-62c1-4f87-b320-80327be23ca9-tls-assets\" (UniqueName: \"kubernetes.io/projected/b24f6847-62c1-4f87-b320-80327be23ca9-tls-assets\") pod \"b24f6847-62c1-4f87-b320-80327be23ca9\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") "
Nov 02 01:26:20.567697 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.567653    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-main-tls\") pod \"b24f6847-62c1-4f87-b320-80327be23ca9\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") "
Nov 02 01:26:20.567697 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.567680    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-web-config\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-web-config\") pod \"b24f6847-62c1-4f87-b320-80327be23ca9\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") "
Nov 02 01:26:20.568323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.567717    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"b24f6847-62c1-4f87-b320-80327be23ca9\" (UID: \"b24f6847-62c1-4f87-b320-80327be23ca9\") "
Nov 02 01:26:20.568323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.567775    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b24f6847-62c1-4f87-b320-80327be23ca9-metrics-client-ca" pod "b24f6847-62c1-4f87-b320-80327be23ca9" (UID: "b24f6847-62c1-4f87-b320-80327be23ca9"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:26:20.568323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.567804    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24f6847-62c1-4f87-b320-80327be23ca9-alertmanager-main-db" pod "b24f6847-62c1-4f87-b320-80327be23ca9" (UID: "b24f6847-62c1-4f87-b320-80327be23ca9"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:26:20.568323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.568211    5639 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b24f6847-62c1-4f87-b320-80327be23ca9-alertmanager-main-db\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:20.568323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.568235    5639 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b24f6847-62c1-4f87-b320-80327be23ca9-metrics-client-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:20.568590 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.568324    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b24f6847-62c1-4f87-b320-80327be23ca9-alertmanager-trusted-ca-bundle" pod "b24f6847-62c1-4f87-b320-80327be23ca9" (UID: "b24f6847-62c1-4f87-b320-80327be23ca9"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:26:20.572674 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.572063    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy-metric" pod "b24f6847-62c1-4f87-b320-80327be23ca9" (UID: "b24f6847-62c1-4f87-b320-80327be23ca9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:26:20.574338 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.573090    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24f6847-62c1-4f87-b320-80327be23ca9-tls-assets" pod "b24f6847-62c1-4f87-b320-80327be23ca9" (UID: "b24f6847-62c1-4f87-b320-80327be23ca9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:26:20.574338 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.573086    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy-web" pod "b24f6847-62c1-4f87-b320-80327be23ca9" (UID: "b24f6847-62c1-4f87-b320-80327be23ca9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:26:20.575924 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.575898    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24f6847-62c1-4f87-b320-80327be23ca9-kube-api-access-bdlxn" pod "b24f6847-62c1-4f87-b320-80327be23ca9" (UID: "b24f6847-62c1-4f87-b320-80327be23ca9"). InnerVolumeSpecName "kube-api-access-bdlxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:26:20.576036 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.575940    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy" pod "b24f6847-62c1-4f87-b320-80327be23ca9" (UID: "b24f6847-62c1-4f87-b320-80327be23ca9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:26:20.576575 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.576238    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24f6847-62c1-4f87-b320-80327be23ca9-config-out" pod "b24f6847-62c1-4f87-b320-80327be23ca9" (UID: "b24f6847-62c1-4f87-b320-80327be23ca9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:26:20.576575 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.576056    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-main-tls" pod "b24f6847-62c1-4f87-b320-80327be23ca9" (UID: "b24f6847-62c1-4f87-b320-80327be23ca9"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:26:20.576575 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.576310    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-config-volume" pod "b24f6847-62c1-4f87-b320-80327be23ca9" (UID: "b24f6847-62c1-4f87-b320-80327be23ca9"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:26:20.579297 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.579260    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-cluster-tls-config" pod "b24f6847-62c1-4f87-b320-80327be23ca9" (UID: "b24f6847-62c1-4f87-b320-80327be23ca9"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:26:20.584684 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.584663    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-web-config" pod "b24f6847-62c1-4f87-b320-80327be23ca9" (UID: "b24f6847-62c1-4f87-b320-80327be23ca9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:26:20.669367 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.669328    5639 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b24f6847-62c1-4f87-b320-80327be23ca9-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:20.669367 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.669359    5639 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:20.669367 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.669371    5639 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:20.669587 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.669380    5639 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:20.669587 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.669390    5639 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b24f6847-62c1-4f87-b320-80327be23ca9-tls-assets\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:20.669587 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.669399    5639 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-main-tls\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:20.669587 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.669407    5639 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-web-config\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:20.669587 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.669416    5639 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:20.669587 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.669426    5639 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b24f6847-62c1-4f87-b320-80327be23ca9-config-out\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:20.669587 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.669434    5639 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b24f6847-62c1-4f87-b320-80327be23ca9-cluster-tls-config\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:20.669587 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:20.669442    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdlxn\" (UniqueName: \"kubernetes.io/projected/b24f6847-62c1-4f87-b320-80327be23ca9-kube-api-access-bdlxn\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:21.443658 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.443627    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b24f6847-62c1-4f87-b320-80327be23ca9","Type":"ContainerDied","Data":"31f315b40def8816fd32f0a2dbb100de956ef67ba6cfea10763fefa9102a5503"}
Nov 02 01:26:21.444116 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.443667    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.444116 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.443708    5639 scope.go:122] "RemoveContainer" containerID="9009b04852505c6f7aecebfd17f4f7b23bc5ee793080eded82aa2c0a36c3da3f"
Nov 02 01:26:21.451093 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.451075    5639 scope.go:122] "RemoveContainer" containerID="2154e062960bb97fa61add68105bf163c102ca16cb7ff631642808e15955dbb8"
Nov 02 01:26:21.457782 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.457762    5639 scope.go:122] "RemoveContainer" containerID="e4bd992cf245d9545be341c555b4e9faaf95c09115677dc2d433a11c19f1b0d5"
Nov 02 01:26:21.464325 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.464309    5639 scope.go:122] "RemoveContainer" containerID="0f4531dc393e10959694446647655bccd77e1e3b837bb5ad6d0764ec41518de3"
Nov 02 01:26:21.472482 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.472389    5639 scope.go:122] "RemoveContainer" containerID="fef375051d2276b6faa3bffc7ee304a8a508397fc33754e4ef3290a121fe0139"
Nov 02 01:26:21.479129 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.479111    5639 scope.go:122] "RemoveContainer" containerID="bf7dd96fc431d213b1729dd5e71942568d39bfdedc3c918b18b3bec8ed1de31f"
Nov 02 01:26:21.485851 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.485833    5639 scope.go:122] "RemoveContainer" containerID="a2f9b1af7a2b352155f273921041bf5082eb862f39ee2a70bd1946b4ae7d4b77"
Nov 02 01:26:21.660050 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.660018    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 01:26:21.709981 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.709948    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 01:26:21.710197 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.710179    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 01:26:21.710467 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.710451    5639 cpu_manager.go:414] "containerMap: removing container" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="init-config-reloader"
Nov 02 01:26:21.710467 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.710465    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="init-config-reloader"
Nov 02 01:26:21.710550 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.710477    5639 cpu_manager.go:414] "containerMap: removing container" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="prom-label-proxy"
Nov 02 01:26:21.710550 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.710482    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="prom-label-proxy"
Nov 02 01:26:21.710550 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.710489    5639 cpu_manager.go:414] "containerMap: removing container" podUID="887b1b53-10f5-4392-89a5-f90a431e60ea" containerName="console"
Nov 02 01:26:21.710550 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.710494    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="887b1b53-10f5-4392-89a5-f90a431e60ea" containerName="console"
Nov 02 01:26:21.710550 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.710508    5639 cpu_manager.go:414] "containerMap: removing container" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="kube-rbac-proxy-metric"
Nov 02 01:26:21.710550 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.710514    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="kube-rbac-proxy-metric"
Nov 02 01:26:21.710550 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.710523    5639 cpu_manager.go:414] "containerMap: removing container" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="kube-rbac-proxy"
Nov 02 01:26:21.710550 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.710527    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="kube-rbac-proxy"
Nov 02 01:26:21.710550 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.710539    5639 cpu_manager.go:414] "containerMap: removing container" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="kube-rbac-proxy-web"
Nov 02 01:26:21.710550 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.710543    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="kube-rbac-proxy-web"
Nov 02 01:26:21.710550 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.710549    5639 cpu_manager.go:414] "containerMap: removing container" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="config-reloader"
Nov 02 01:26:21.710550 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.710553    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="config-reloader"
Nov 02 01:26:21.710878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.710558    5639 cpu_manager.go:414] "containerMap: removing container" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="alertmanager"
Nov 02 01:26:21.710878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.710562    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="alertmanager"
Nov 02 01:26:21.710878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.710605    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="kube-rbac-proxy-metric"
Nov 02 01:26:21.710878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.710613    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="kube-rbac-proxy"
Nov 02 01:26:21.710878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.710619    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="kube-rbac-proxy-web"
Nov 02 01:26:21.710878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.710624    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="config-reloader"
Nov 02 01:26:21.710878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.710629    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="alertmanager"
Nov 02 01:26:21.710878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.710636    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="init-config-reloader"
Nov 02 01:26:21.710878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.710641    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" containerName="prom-label-proxy"
Nov 02 01:26:21.710878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.710646    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="887b1b53-10f5-4392-89a5-f90a431e60ea" containerName="console"
Nov 02 01:26:21.714064 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.714046    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.767075 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.766986    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 02 01:26:21.767075 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.767005    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jpr5g\""
Nov 02 01:26:21.767075 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.767026    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 02 01:26:21.767342 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.767090    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 02 01:26:21.767342 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.767108    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 02 01:26:21.767342 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.767114    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 02 01:26:21.767342 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.767123    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 02 01:26:21.767342 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.767123    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 02 01:26:21.767342 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.767125    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 02 01:26:21.778438 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.778404    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8aa5a540-cf10-47bc-954c-a92d8418f7e7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.778551 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.778448    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8aa5a540-cf10-47bc-954c-a92d8418f7e7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.778551 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.778527    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa5a540-cf10-47bc-954c-a92d8418f7e7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.778551 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.778546    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk4jz\" (UniqueName: \"kubernetes.io/projected/8aa5a540-cf10-47bc-954c-a92d8418f7e7-kube-api-access-nk4jz\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.778647 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.778565    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8aa5a540-cf10-47bc-954c-a92d8418f7e7-config-volume\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.778647 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.778589    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8aa5a540-cf10-47bc-954c-a92d8418f7e7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.778703 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.778660    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5a540-cf10-47bc-954c-a92d8418f7e7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.778735 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.778699    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8aa5a540-cf10-47bc-954c-a92d8418f7e7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.778767 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.778751    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8aa5a540-cf10-47bc-954c-a92d8418f7e7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.778799 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.778782    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8aa5a540-cf10-47bc-954c-a92d8418f7e7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.778828 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.778803    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8aa5a540-cf10-47bc-954c-a92d8418f7e7-web-config\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.778828 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.778821    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8aa5a540-cf10-47bc-954c-a92d8418f7e7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.778887 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.778878    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8aa5a540-cf10-47bc-954c-a92d8418f7e7-config-out\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.816544 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.816510    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 02 01:26:21.879769 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.879739    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8aa5a540-cf10-47bc-954c-a92d8418f7e7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.879769 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.879777    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa5a540-cf10-47bc-954c-a92d8418f7e7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.879978 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.879794    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk4jz\" (UniqueName: \"kubernetes.io/projected/8aa5a540-cf10-47bc-954c-a92d8418f7e7-kube-api-access-nk4jz\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.879978 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.879809    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8aa5a540-cf10-47bc-954c-a92d8418f7e7-config-volume\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.879978 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.879835    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8aa5a540-cf10-47bc-954c-a92d8418f7e7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.879978 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.879871    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5a540-cf10-47bc-954c-a92d8418f7e7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.879978 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.879899    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8aa5a540-cf10-47bc-954c-a92d8418f7e7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.879978 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.879926    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8aa5a540-cf10-47bc-954c-a92d8418f7e7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.879978 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.879956    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8aa5a540-cf10-47bc-954c-a92d8418f7e7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.879978 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.879975    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8aa5a540-cf10-47bc-954c-a92d8418f7e7-web-config\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.880416 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.879991    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8aa5a540-cf10-47bc-954c-a92d8418f7e7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.880416 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.880015    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8aa5a540-cf10-47bc-954c-a92d8418f7e7-config-out\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.880416 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.880049    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8aa5a540-cf10-47bc-954c-a92d8418f7e7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.880416 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.880252    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8aa5a540-cf10-47bc-954c-a92d8418f7e7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.880416 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.880317    5639 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 01:26:21.880682 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.880658    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa5a540-cf10-47bc-954c-a92d8418f7e7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.880867 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.880836    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5a540-cf10-47bc-954c-a92d8418f7e7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.883174 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.883145    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8aa5a540-cf10-47bc-954c-a92d8418f7e7-config-out\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.883307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.883246    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8aa5a540-cf10-47bc-954c-a92d8418f7e7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.883619 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.883574    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8aa5a540-cf10-47bc-954c-a92d8418f7e7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.883713 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.883615    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8aa5a540-cf10-47bc-954c-a92d8418f7e7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.883713 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.883673    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8aa5a540-cf10-47bc-954c-a92d8418f7e7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.883892 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.883870    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8aa5a540-cf10-47bc-954c-a92d8418f7e7-config-volume\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.883962 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.883942    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8aa5a540-cf10-47bc-954c-a92d8418f7e7-web-config\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.884271 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.884251    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8aa5a540-cf10-47bc-954c-a92d8418f7e7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.885294 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.885258    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8aa5a540-cf10-47bc-954c-a92d8418f7e7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.941132 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.941098    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk4jz\" (UniqueName: \"kubernetes.io/projected/8aa5a540-cf10-47bc-954c-a92d8418f7e7-kube-api-access-nk4jz\") pod \"alertmanager-main-0\" (UID: \"8aa5a540-cf10-47bc-954c-a92d8418f7e7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:21.950756 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:21.950716    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 01:26:22.028921 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:22.028840    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:26:22.455393 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:22.455356    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 01:26:22.458709 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:26:22.458680    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa5a540_cf10_47bc_954c_a92d8418f7e7.slice/crio-33937baa0565640d8b4a2ed21d1a456674ea86b56db74359e60e9bfae298f50b WatchSource:0}: Error finding container 33937baa0565640d8b4a2ed21d1a456674ea86b56db74359e60e9bfae298f50b: Status 404 returned error can't find the container with id 33937baa0565640d8b4a2ed21d1a456674ea86b56db74359e60e9bfae298f50b
Nov 02 01:26:22.956046 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:22.956015    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"]
Nov 02 01:26:22.958185 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:22.958156    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"
Nov 02 01:26:23.010745 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.010708    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 02 01:26:23.010745 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.010713    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 02 01:26:23.010982 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.010781    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 02 01:26:23.010982 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.010793    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 02 01:26:23.010982 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.010901    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 02 01:26:23.010982 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.010970    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-fzdm9\""
Nov 02 01:26:23.090241 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.090210    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/27bd836b-2ab9-462c-8e6d-63fe3d090d04-secret-telemeter-client\") pod \"telemeter-client-7bdd7cc4cc-566sl\" (UID: \"27bd836b-2ab9-462c-8e6d-63fe3d090d04\") " pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"
Nov 02 01:26:23.090420 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.090269    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/27bd836b-2ab9-462c-8e6d-63fe3d090d04-federate-client-tls\") pod \"telemeter-client-7bdd7cc4cc-566sl\" (UID: \"27bd836b-2ab9-462c-8e6d-63fe3d090d04\") " pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"
Nov 02 01:26:23.090420 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.090364    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27bd836b-2ab9-462c-8e6d-63fe3d090d04-serving-certs-ca-bundle\") pod \"telemeter-client-7bdd7cc4cc-566sl\" (UID: \"27bd836b-2ab9-462c-8e6d-63fe3d090d04\") " pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"
Nov 02 01:26:23.090491 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.090430    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27bd836b-2ab9-462c-8e6d-63fe3d090d04-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7bdd7cc4cc-566sl\" (UID: \"27bd836b-2ab9-462c-8e6d-63fe3d090d04\") " pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"
Nov 02 01:26:23.090491 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.090469    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27bd836b-2ab9-462c-8e6d-63fe3d090d04-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7bdd7cc4cc-566sl\" (UID: \"27bd836b-2ab9-462c-8e6d-63fe3d090d04\") " pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"
Nov 02 01:26:23.090546 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.090490    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brnfp\" (UniqueName: \"kubernetes.io/projected/27bd836b-2ab9-462c-8e6d-63fe3d090d04-kube-api-access-brnfp\") pod \"telemeter-client-7bdd7cc4cc-566sl\" (UID: \"27bd836b-2ab9-462c-8e6d-63fe3d090d04\") " pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"
Nov 02 01:26:23.090546 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.090509    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/27bd836b-2ab9-462c-8e6d-63fe3d090d04-telemeter-client-tls\") pod \"telemeter-client-7bdd7cc4cc-566sl\" (UID: \"27bd836b-2ab9-462c-8e6d-63fe3d090d04\") " pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"
Nov 02 01:26:23.090648 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.090590    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27bd836b-2ab9-462c-8e6d-63fe3d090d04-metrics-client-ca\") pod \"telemeter-client-7bdd7cc4cc-566sl\" (UID: \"27bd836b-2ab9-462c-8e6d-63fe3d090d04\") " pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"
Nov 02 01:26:23.110166 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.110128    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 02 01:26:23.110892 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.110871    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"]
Nov 02 01:26:23.191336 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.191296    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27bd836b-2ab9-462c-8e6d-63fe3d090d04-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7bdd7cc4cc-566sl\" (UID: \"27bd836b-2ab9-462c-8e6d-63fe3d090d04\") " pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"
Nov 02 01:26:23.191512 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.191340    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27bd836b-2ab9-462c-8e6d-63fe3d090d04-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7bdd7cc4cc-566sl\" (UID: \"27bd836b-2ab9-462c-8e6d-63fe3d090d04\") " pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"
Nov 02 01:26:23.191512 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.191371    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brnfp\" (UniqueName: \"kubernetes.io/projected/27bd836b-2ab9-462c-8e6d-63fe3d090d04-kube-api-access-brnfp\") pod \"telemeter-client-7bdd7cc4cc-566sl\" (UID: \"27bd836b-2ab9-462c-8e6d-63fe3d090d04\") " pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"
Nov 02 01:26:23.191512 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.191395    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/27bd836b-2ab9-462c-8e6d-63fe3d090d04-telemeter-client-tls\") pod \"telemeter-client-7bdd7cc4cc-566sl\" (UID: \"27bd836b-2ab9-462c-8e6d-63fe3d090d04\") " pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"
Nov 02 01:26:23.191512 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.191445    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27bd836b-2ab9-462c-8e6d-63fe3d090d04-metrics-client-ca\") pod \"telemeter-client-7bdd7cc4cc-566sl\" (UID: \"27bd836b-2ab9-462c-8e6d-63fe3d090d04\") " pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"
Nov 02 01:26:23.191512 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.191487    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/27bd836b-2ab9-462c-8e6d-63fe3d090d04-secret-telemeter-client\") pod \"telemeter-client-7bdd7cc4cc-566sl\" (UID: \"27bd836b-2ab9-462c-8e6d-63fe3d090d04\") " pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"
Nov 02 01:26:23.191763 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.191549    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/27bd836b-2ab9-462c-8e6d-63fe3d090d04-federate-client-tls\") pod \"telemeter-client-7bdd7cc4cc-566sl\" (UID: \"27bd836b-2ab9-462c-8e6d-63fe3d090d04\") " pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"
Nov 02 01:26:23.191763 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.191579    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27bd836b-2ab9-462c-8e6d-63fe3d090d04-serving-certs-ca-bundle\") pod \"telemeter-client-7bdd7cc4cc-566sl\" (UID: \"27bd836b-2ab9-462c-8e6d-63fe3d090d04\") " pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"
Nov 02 01:26:23.192327 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.192236    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27bd836b-2ab9-462c-8e6d-63fe3d090d04-metrics-client-ca\") pod \"telemeter-client-7bdd7cc4cc-566sl\" (UID: \"27bd836b-2ab9-462c-8e6d-63fe3d090d04\") " pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"
Nov 02 01:26:23.192477 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.192452    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27bd836b-2ab9-462c-8e6d-63fe3d090d04-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7bdd7cc4cc-566sl\" (UID: \"27bd836b-2ab9-462c-8e6d-63fe3d090d04\") " pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"
Nov 02 01:26:23.192601 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.192572    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27bd836b-2ab9-462c-8e6d-63fe3d090d04-serving-certs-ca-bundle\") pod \"telemeter-client-7bdd7cc4cc-566sl\" (UID: \"27bd836b-2ab9-462c-8e6d-63fe3d090d04\") " pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"
Nov 02 01:26:23.194049 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.194030    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/27bd836b-2ab9-462c-8e6d-63fe3d090d04-telemeter-client-tls\") pod \"telemeter-client-7bdd7cc4cc-566sl\" (UID: \"27bd836b-2ab9-462c-8e6d-63fe3d090d04\") " pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"
Nov 02 01:26:23.194126 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.194060    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27bd836b-2ab9-462c-8e6d-63fe3d090d04-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7bdd7cc4cc-566sl\" (UID: \"27bd836b-2ab9-462c-8e6d-63fe3d090d04\") " pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"
Nov 02 01:26:23.194549 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.194528    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/27bd836b-2ab9-462c-8e6d-63fe3d090d04-secret-telemeter-client\") pod \"telemeter-client-7bdd7cc4cc-566sl\" (UID: \"27bd836b-2ab9-462c-8e6d-63fe3d090d04\") " pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"
Nov 02 01:26:23.194621 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.194560    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/27bd836b-2ab9-462c-8e6d-63fe3d090d04-federate-client-tls\") pod \"telemeter-client-7bdd7cc4cc-566sl\" (UID: \"27bd836b-2ab9-462c-8e6d-63fe3d090d04\") " pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"
Nov 02 01:26:23.215772 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.215711    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b24f6847-62c1-4f87-b320-80327be23ca9" path="/var/lib/kubelet/pods/b24f6847-62c1-4f87-b320-80327be23ca9/volumes"
Nov 02 01:26:23.252370 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.252345    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brnfp\" (UniqueName: \"kubernetes.io/projected/27bd836b-2ab9-462c-8e6d-63fe3d090d04-kube-api-access-brnfp\") pod \"telemeter-client-7bdd7cc4cc-566sl\" (UID: \"27bd836b-2ab9-462c-8e6d-63fe3d090d04\") " pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"
Nov 02 01:26:23.270662 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.270639    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"
Nov 02 01:26:23.454048 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.454013    5639 generic.go:356] "Generic (PLEG): container finished" podID="8aa5a540-cf10-47bc-954c-a92d8418f7e7" containerID="69024b0b7413d81ff78cca8fb56dfb99d1e7b47713cb82360f034fa9956f4301" exitCode=0
Nov 02 01:26:23.454222 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.454101    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8aa5a540-cf10-47bc-954c-a92d8418f7e7","Type":"ContainerDied","Data":"69024b0b7413d81ff78cca8fb56dfb99d1e7b47713cb82360f034fa9956f4301"}
Nov 02 01:26:23.454222 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.454161    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8aa5a540-cf10-47bc-954c-a92d8418f7e7","Type":"ContainerStarted","Data":"33937baa0565640d8b4a2ed21d1a456674ea86b56db74359e60e9bfae298f50b"}
Nov 02 01:26:23.699138 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.699112    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl"]
Nov 02 01:26:23.704564 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:26:23.704529    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27bd836b_2ab9_462c_8e6d_63fe3d090d04.slice/crio-7c3afc88976ed06187db20530c5496c02a4597e2c74aff65b781a58295f9c9b5 WatchSource:0}: Error finding container 7c3afc88976ed06187db20530c5496c02a4597e2c74aff65b781a58295f9c9b5: Status 404 returned error can't find the container with id 7c3afc88976ed06187db20530c5496c02a4597e2c74aff65b781a58295f9c9b5
Nov 02 01:26:23.706675 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:23.706659    5639 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:26:24.461189 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:24.461150    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8aa5a540-cf10-47bc-954c-a92d8418f7e7","Type":"ContainerStarted","Data":"1d93afff6b1a8f48b5a514441dd66b30f31545e334c96c61573713d0d2015915"}
Nov 02 01:26:24.461400 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:24.461230    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8aa5a540-cf10-47bc-954c-a92d8418f7e7","Type":"ContainerStarted","Data":"17bbf94d94ead3f490ebf76407cef809558fdc8ca30ad07d95432b6afb8bb300"}
Nov 02 01:26:24.461400 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:24.461268    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8aa5a540-cf10-47bc-954c-a92d8418f7e7","Type":"ContainerStarted","Data":"1d1338041d1c6929849a6e0a76d3b9902ff3a101f9c1056c2fe48ddc9b0f8311"}
Nov 02 01:26:24.461400 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:24.461320    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8aa5a540-cf10-47bc-954c-a92d8418f7e7","Type":"ContainerStarted","Data":"ab8906deadb27f05047927e6a8fe280f314abd15ef9ac4d205ec6a290f3c5c75"}
Nov 02 01:26:24.461400 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:24.461345    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8aa5a540-cf10-47bc-954c-a92d8418f7e7","Type":"ContainerStarted","Data":"149e31b7d8d6d48d9c9cf7d4935419c16323a14402cbc0e9232e52e96625ecd7"}
Nov 02 01:26:24.461400 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:24.461373    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8aa5a540-cf10-47bc-954c-a92d8418f7e7","Type":"ContainerStarted","Data":"a3f7e3155b8c3a17c6d10ec196833937a1927fba6cb08035c5a488e44fd89a98"}
Nov 02 01:26:24.462446 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:24.462423    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl" event={"ID":"27bd836b-2ab9-462c-8e6d-63fe3d090d04","Type":"ContainerStarted","Data":"7c3afc88976ed06187db20530c5496c02a4597e2c74aff65b781a58295f9c9b5"}
Nov 02 01:26:24.590987 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:24.590933    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.578331189 podStartE2EDuration="3.590919008s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.587819ms" isStatefulPod=false podCreationTimestamp="2026-11-02 01:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:26:24.589980627 +0000 UTC m=+49.493662741" watchObservedRunningTime="2026-11-02 01:26:24.590919008 +0000 UTC m=+49.494601121"
Nov 02 01:26:26.471962 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:26.471869    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl" event={"ID":"27bd836b-2ab9-462c-8e6d-63fe3d090d04","Type":"ContainerStarted","Data":"8283fa8edf72f281c2f0962be1b04df62d733689954edd9003292763032fee08"}
Nov 02 01:26:26.472378 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:26.471963    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl" event={"ID":"27bd836b-2ab9-462c-8e6d-63fe3d090d04","Type":"ContainerStarted","Data":"6e2f50c68f3d230346e2c526b3288fd6179d59be6ff7274fe1c62fd8c0b26349"}
Nov 02 01:26:26.472378 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:26.471986    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl" event={"ID":"27bd836b-2ab9-462c-8e6d-63fe3d090d04","Type":"ContainerStarted","Data":"2e59d929728249995002bb1e7634c646de01a71c50f9726b65dfcbed035a4377"}
Nov 02 01:26:26.740982 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:26.740869    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7bdd7cc4cc-566sl" podStartSLOduration=2.353044803 podStartE2EDuration="4.740857195s" totalImagesPullingTime="2.387812392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:26:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:26:26.691247985 +0000 UTC m=+51.594930101" watchObservedRunningTime="2026-11-02 01:26:26.740857195 +0000 UTC m=+51.644539308"
Nov 02 01:26:27.164039 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.164002    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-cdff6c64f-m9flm"]
Nov 02 01:26:27.166044 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.166022    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:27.218874 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.218824    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 02 01:26:27.219022 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.218988    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 02 01:26:27.219058 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.219042    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lnsht\""
Nov 02 01:26:27.219335 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.219318    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 02 01:26:27.219335 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.219329    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 02 01:26:27.219421 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.219329    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 02 01:26:27.219421 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.219318    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 02 01:26:27.219421 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.219370    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 02 01:26:27.226026 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.226003    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-oauth-serving-cert\") pod \"console-cdff6c64f-m9flm\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") " pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:27.226133 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.226037    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-service-ca\") pod \"console-cdff6c64f-m9flm\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") " pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:27.226133 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.226074    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62de3501-f48f-4f6b-a660-9df422b21bbf-tmp\") pod \"console-cdff6c64f-m9flm\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") " pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:27.226238 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.226219    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62de3501-f48f-4f6b-a660-9df422b21bbf-console-serving-cert\") pod \"console-cdff6c64f-m9flm\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") " pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:27.226320 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.226302    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spxhg\" (UniqueName: \"kubernetes.io/projected/62de3501-f48f-4f6b-a660-9df422b21bbf-kube-api-access-spxhg\") pod \"console-cdff6c64f-m9flm\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") " pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:27.226374 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.226352    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-trusted-ca-bundle\") pod \"console-cdff6c64f-m9flm\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") " pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:27.226440 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.226409    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62de3501-f48f-4f6b-a660-9df422b21bbf-console-oauth-config\") pod \"console-cdff6c64f-m9flm\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") " pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:27.226498 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.226447    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-console-config\") pod \"console-cdff6c64f-m9flm\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") " pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:27.318852 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.318821    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 02 01:26:27.319514 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.319491    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cdff6c64f-m9flm"]
Nov 02 01:26:27.327233 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.327217    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-trusted-ca-bundle\") pod \"console-cdff6c64f-m9flm\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") " pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:27.327299 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.327242    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62de3501-f48f-4f6b-a660-9df422b21bbf-console-oauth-config\") pod \"console-cdff6c64f-m9flm\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") " pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:27.327299 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.327261    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-console-config\") pod \"console-cdff6c64f-m9flm\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") " pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:27.327431 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.327412    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-oauth-serving-cert\") pod \"console-cdff6c64f-m9flm\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") " pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:27.327495 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.327462    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-service-ca\") pod \"console-cdff6c64f-m9flm\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") " pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:27.327548 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.327517    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62de3501-f48f-4f6b-a660-9df422b21bbf-tmp\") pod \"console-cdff6c64f-m9flm\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") " pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:27.327600 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.327563    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62de3501-f48f-4f6b-a660-9df422b21bbf-console-serving-cert\") pod \"console-cdff6c64f-m9flm\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") " pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:27.327725 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.327693    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spxhg\" (UniqueName: \"kubernetes.io/projected/62de3501-f48f-4f6b-a660-9df422b21bbf-kube-api-access-spxhg\") pod \"console-cdff6c64f-m9flm\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") " pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:27.328252 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.328227    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-oauth-serving-cert\") pod \"console-cdff6c64f-m9flm\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") " pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:27.328252 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.328243    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-service-ca\") pod \"console-cdff6c64f-m9flm\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") " pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:27.328407 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.328355    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-trusted-ca-bundle\") pod \"console-cdff6c64f-m9flm\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") " pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:27.328407 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.328353    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-console-config\") pod \"console-cdff6c64f-m9flm\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") " pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:27.328407 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.328391    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62de3501-f48f-4f6b-a660-9df422b21bbf-tmp\") pod \"console-cdff6c64f-m9flm\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") " pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:27.329830 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.329806    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62de3501-f48f-4f6b-a660-9df422b21bbf-console-oauth-config\") pod \"console-cdff6c64f-m9flm\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") " pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:27.329964 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.329948    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62de3501-f48f-4f6b-a660-9df422b21bbf-console-serving-cert\") pod \"console-cdff6c64f-m9flm\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") " pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:27.388872 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.388839    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spxhg\" (UniqueName: \"kubernetes.io/projected/62de3501-f48f-4f6b-a660-9df422b21bbf-kube-api-access-spxhg\") pod \"console-cdff6c64f-m9flm\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") " pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:27.419245 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:26:27.419172    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:26:27.475602 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.475578    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:27.846051 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:27.846024    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cdff6c64f-m9flm"]
Nov 02 01:26:27.849108 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:26:27.849077    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62de3501_f48f_4f6b_a660_9df422b21bbf.slice/crio-038c49c5ced4963014a118738594943c668db5a881b00cc579a5a7530ba296f3 WatchSource:0}: Error finding container 038c49c5ced4963014a118738594943c668db5a881b00cc579a5a7530ba296f3: Status 404 returned error can't find the container with id 038c49c5ced4963014a118738594943c668db5a881b00cc579a5a7530ba296f3
Nov 02 01:26:28.485174 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:28.485139    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cdff6c64f-m9flm" event={"ID":"62de3501-f48f-4f6b-a660-9df422b21bbf","Type":"ContainerStarted","Data":"73799674a1eec4df5c7cbd38b0868787bb18a63072a7db317fd543b6fd75c137"}
Nov 02 01:26:28.485571 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:28.485210    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cdff6c64f-m9flm" event={"ID":"62de3501-f48f-4f6b-a660-9df422b21bbf","Type":"ContainerStarted","Data":"038c49c5ced4963014a118738594943c668db5a881b00cc579a5a7530ba296f3"}
Nov 02 01:26:28.603583 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:28.603527    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-cdff6c64f-m9flm" podStartSLOduration=1.6035145229999999 podStartE2EDuration="1.603514523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:26:28.603169123 +0000 UTC m=+53.506851241" watchObservedRunningTime="2026-11-02 01:26:28.603514523 +0000 UTC m=+53.507196647"
Nov 02 01:26:37.475728 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:37.475690    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:37.476116 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:37.475770    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:37.480638 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:37.480615    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:37.518575 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:26:37.518551    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:26:57.418683 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:26:57.418642    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:27:27.418406 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:27:27.418320    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:27:38.722744 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.722713    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-c55bd6f95-tn8xn"]
Nov 02 01:27:38.726196 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.726174    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:38.785732 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.785695    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c55bd6f95-tn8xn"]
Nov 02 01:27:38.812213 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.812178    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/561b66dc-47f2-491b-b73b-b7691189d9f7-oauth-serving-cert\") pod \"console-c55bd6f95-tn8xn\" (UID: \"561b66dc-47f2-491b-b73b-b7691189d9f7\") " pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:38.812394 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.812231    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/561b66dc-47f2-491b-b73b-b7691189d9f7-console-oauth-config\") pod \"console-c55bd6f95-tn8xn\" (UID: \"561b66dc-47f2-491b-b73b-b7691189d9f7\") " pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:38.812394 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.812317    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/561b66dc-47f2-491b-b73b-b7691189d9f7-tmp\") pod \"console-c55bd6f95-tn8xn\" (UID: \"561b66dc-47f2-491b-b73b-b7691189d9f7\") " pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:38.812394 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.812357    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/561b66dc-47f2-491b-b73b-b7691189d9f7-console-serving-cert\") pod \"console-c55bd6f95-tn8xn\" (UID: \"561b66dc-47f2-491b-b73b-b7691189d9f7\") " pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:38.812495 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.812430    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/561b66dc-47f2-491b-b73b-b7691189d9f7-trusted-ca-bundle\") pod \"console-c55bd6f95-tn8xn\" (UID: \"561b66dc-47f2-491b-b73b-b7691189d9f7\") " pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:38.812495 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.812456    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/561b66dc-47f2-491b-b73b-b7691189d9f7-service-ca\") pod \"console-c55bd6f95-tn8xn\" (UID: \"561b66dc-47f2-491b-b73b-b7691189d9f7\") " pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:38.812495 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.812473    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktj5n\" (UniqueName: \"kubernetes.io/projected/561b66dc-47f2-491b-b73b-b7691189d9f7-kube-api-access-ktj5n\") pod \"console-c55bd6f95-tn8xn\" (UID: \"561b66dc-47f2-491b-b73b-b7691189d9f7\") " pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:38.812495 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.812492    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/561b66dc-47f2-491b-b73b-b7691189d9f7-console-config\") pod \"console-c55bd6f95-tn8xn\" (UID: \"561b66dc-47f2-491b-b73b-b7691189d9f7\") " pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:38.912937 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.912899    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/561b66dc-47f2-491b-b73b-b7691189d9f7-oauth-serving-cert\") pod \"console-c55bd6f95-tn8xn\" (UID: \"561b66dc-47f2-491b-b73b-b7691189d9f7\") " pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:38.913118 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.912955    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/561b66dc-47f2-491b-b73b-b7691189d9f7-console-oauth-config\") pod \"console-c55bd6f95-tn8xn\" (UID: \"561b66dc-47f2-491b-b73b-b7691189d9f7\") " pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:38.913118 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.912993    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/561b66dc-47f2-491b-b73b-b7691189d9f7-tmp\") pod \"console-c55bd6f95-tn8xn\" (UID: \"561b66dc-47f2-491b-b73b-b7691189d9f7\") " pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:38.913118 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.913014    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/561b66dc-47f2-491b-b73b-b7691189d9f7-console-serving-cert\") pod \"console-c55bd6f95-tn8xn\" (UID: \"561b66dc-47f2-491b-b73b-b7691189d9f7\") " pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:38.913118 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.913047    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/561b66dc-47f2-491b-b73b-b7691189d9f7-trusted-ca-bundle\") pod \"console-c55bd6f95-tn8xn\" (UID: \"561b66dc-47f2-491b-b73b-b7691189d9f7\") " pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:38.913118 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.913076    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/561b66dc-47f2-491b-b73b-b7691189d9f7-service-ca\") pod \"console-c55bd6f95-tn8xn\" (UID: \"561b66dc-47f2-491b-b73b-b7691189d9f7\") " pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:38.913118 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.913098    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktj5n\" (UniqueName: \"kubernetes.io/projected/561b66dc-47f2-491b-b73b-b7691189d9f7-kube-api-access-ktj5n\") pod \"console-c55bd6f95-tn8xn\" (UID: \"561b66dc-47f2-491b-b73b-b7691189d9f7\") " pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:38.913456 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.913121    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/561b66dc-47f2-491b-b73b-b7691189d9f7-console-config\") pod \"console-c55bd6f95-tn8xn\" (UID: \"561b66dc-47f2-491b-b73b-b7691189d9f7\") " pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:38.913694 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.913663    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/561b66dc-47f2-491b-b73b-b7691189d9f7-tmp\") pod \"console-c55bd6f95-tn8xn\" (UID: \"561b66dc-47f2-491b-b73b-b7691189d9f7\") " pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:38.913838 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.913777    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/561b66dc-47f2-491b-b73b-b7691189d9f7-oauth-serving-cert\") pod \"console-c55bd6f95-tn8xn\" (UID: \"561b66dc-47f2-491b-b73b-b7691189d9f7\") " pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:38.913950 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.913923    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/561b66dc-47f2-491b-b73b-b7691189d9f7-service-ca\") pod \"console-c55bd6f95-tn8xn\" (UID: \"561b66dc-47f2-491b-b73b-b7691189d9f7\") " pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:38.914036 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.913953    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/561b66dc-47f2-491b-b73b-b7691189d9f7-trusted-ca-bundle\") pod \"console-c55bd6f95-tn8xn\" (UID: \"561b66dc-47f2-491b-b73b-b7691189d9f7\") " pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:38.914036 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.913980    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/561b66dc-47f2-491b-b73b-b7691189d9f7-console-config\") pod \"console-c55bd6f95-tn8xn\" (UID: \"561b66dc-47f2-491b-b73b-b7691189d9f7\") " pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:38.916219 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.916194    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/561b66dc-47f2-491b-b73b-b7691189d9f7-console-oauth-config\") pod \"console-c55bd6f95-tn8xn\" (UID: \"561b66dc-47f2-491b-b73b-b7691189d9f7\") " pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:38.916358 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.916340    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/561b66dc-47f2-491b-b73b-b7691189d9f7-console-serving-cert\") pod \"console-c55bd6f95-tn8xn\" (UID: \"561b66dc-47f2-491b-b73b-b7691189d9f7\") " pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:38.974704 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:38.974625    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktj5n\" (UniqueName: \"kubernetes.io/projected/561b66dc-47f2-491b-b73b-b7691189d9f7-kube-api-access-ktj5n\") pod \"console-c55bd6f95-tn8xn\" (UID: \"561b66dc-47f2-491b-b73b-b7691189d9f7\") " pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:39.037560 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:39.037521    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:39.407626 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:39.407600    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c55bd6f95-tn8xn"]
Nov 02 01:27:39.410788 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:27:39.410752    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod561b66dc_47f2_491b_b73b_b7691189d9f7.slice/crio-bae8c53f73f07145fabf7e728a13e05c1e235f18dcf0f230f8c80f5b77b9523c WatchSource:0}: Error finding container bae8c53f73f07145fabf7e728a13e05c1e235f18dcf0f230f8c80f5b77b9523c: Status 404 returned error can't find the container with id bae8c53f73f07145fabf7e728a13e05c1e235f18dcf0f230f8c80f5b77b9523c
Nov 02 01:27:39.730368 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:39.730257    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c55bd6f95-tn8xn" event={"ID":"561b66dc-47f2-491b-b73b-b7691189d9f7","Type":"ContainerStarted","Data":"936c3da69db1bd34c5c946ff13da7d84d73bcb9e94e92dcff639c86c96ce74cb"}
Nov 02 01:27:39.730738 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:39.730370    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c55bd6f95-tn8xn" event={"ID":"561b66dc-47f2-491b-b73b-b7691189d9f7","Type":"ContainerStarted","Data":"bae8c53f73f07145fabf7e728a13e05c1e235f18dcf0f230f8c80f5b77b9523c"}
Nov 02 01:27:39.849681 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:39.849628    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-c55bd6f95-tn8xn" podStartSLOduration=1.84961409 podStartE2EDuration="1.84961409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:39.848696073 +0000 UTC m=+124.752378190" watchObservedRunningTime="2026-11-02 01:27:39.84961409 +0000 UTC m=+124.753296194"
Nov 02 01:27:49.038456 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:49.038392    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:49.038456 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:49.038465    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:49.043149 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:49.043127    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:49.770399 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:49.770370    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c55bd6f95-tn8xn"
Nov 02 01:27:49.939325 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:27:49.939291    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cdff6c64f-m9flm"]
Nov 02 01:27:57.419084 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:27:57.419046    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:28:14.959172 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:14.959109    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-cdff6c64f-m9flm" podUID="62de3501-f48f-4f6b-a660-9df422b21bbf" containerName="console" containerID="cri-o://73799674a1eec4df5c7cbd38b0868787bb18a63072a7db317fd543b6fd75c137" gracePeriod=15
Nov 02 01:28:15.200705 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.200679    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:28:15.292815 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.292719    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-service-ca\" (UniqueName: \"kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-service-ca\") pod \"62de3501-f48f-4f6b-a660-9df422b21bbf\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") "
Nov 02 01:28:15.292815 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.292798    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62de3501-f48f-4f6b-a660-9df422b21bbf-tmp\" (UniqueName: \"kubernetes.io/empty-dir/62de3501-f48f-4f6b-a660-9df422b21bbf-tmp\") pod \"62de3501-f48f-4f6b-a660-9df422b21bbf\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") "
Nov 02 01:28:15.293043 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.292838    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62de3501-f48f-4f6b-a660-9df422b21bbf-kube-api-access-spxhg\" (UniqueName: \"kubernetes.io/projected/62de3501-f48f-4f6b-a660-9df422b21bbf-kube-api-access-spxhg\") pod \"62de3501-f48f-4f6b-a660-9df422b21bbf\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") "
Nov 02 01:28:15.293043 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.292870    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-console-config\" (UniqueName: \"kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-console-config\") pod \"62de3501-f48f-4f6b-a660-9df422b21bbf\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") "
Nov 02 01:28:15.293043 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.292917    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62de3501-f48f-4f6b-a660-9df422b21bbf-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62de3501-f48f-4f6b-a660-9df422b21bbf-console-oauth-config\") pod \"62de3501-f48f-4f6b-a660-9df422b21bbf\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") "
Nov 02 01:28:15.293043 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.292976    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-oauth-serving-cert\") pod \"62de3501-f48f-4f6b-a660-9df422b21bbf\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") "
Nov 02 01:28:15.293043 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.293007    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62de3501-f48f-4f6b-a660-9df422b21bbf-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62de3501-f48f-4f6b-a660-9df422b21bbf-console-serving-cert\") pod \"62de3501-f48f-4f6b-a660-9df422b21bbf\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") "
Nov 02 01:28:15.293043 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.293037    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-trusted-ca-bundle\") pod \"62de3501-f48f-4f6b-a660-9df422b21bbf\" (UID: \"62de3501-f48f-4f6b-a660-9df422b21bbf\") "
Nov 02 01:28:15.293336 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.293100    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62de3501-f48f-4f6b-a660-9df422b21bbf-tmp" pod "62de3501-f48f-4f6b-a660-9df422b21bbf" (UID: "62de3501-f48f-4f6b-a660-9df422b21bbf"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:28:15.293336 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.293116    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-service-ca" pod "62de3501-f48f-4f6b-a660-9df422b21bbf" (UID: "62de3501-f48f-4f6b-a660-9df422b21bbf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:28:15.293438 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.293417    5639 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-service-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:28:15.293498 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.293447    5639 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62de3501-f48f-4f6b-a660-9df422b21bbf-tmp\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:28:15.293498 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.293457    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-oauth-serving-cert" pod "62de3501-f48f-4f6b-a660-9df422b21bbf" (UID: "62de3501-f48f-4f6b-a660-9df422b21bbf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:28:15.293614 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.293574    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-trusted-ca-bundle" pod "62de3501-f48f-4f6b-a660-9df422b21bbf" (UID: "62de3501-f48f-4f6b-a660-9df422b21bbf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:28:15.293614 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.293602    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-console-config" pod "62de3501-f48f-4f6b-a660-9df422b21bbf" (UID: "62de3501-f48f-4f6b-a660-9df422b21bbf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:28:15.295231 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.295200    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62de3501-f48f-4f6b-a660-9df422b21bbf-kube-api-access-spxhg" pod "62de3501-f48f-4f6b-a660-9df422b21bbf" (UID: "62de3501-f48f-4f6b-a660-9df422b21bbf"). InnerVolumeSpecName "kube-api-access-spxhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:28:15.295231 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.295219    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62de3501-f48f-4f6b-a660-9df422b21bbf-console-serving-cert" pod "62de3501-f48f-4f6b-a660-9df422b21bbf" (UID: "62de3501-f48f-4f6b-a660-9df422b21bbf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:28:15.295406 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.295266    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62de3501-f48f-4f6b-a660-9df422b21bbf-console-oauth-config" pod "62de3501-f48f-4f6b-a660-9df422b21bbf" (UID: "62de3501-f48f-4f6b-a660-9df422b21bbf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:28:15.394656 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.394611    5639 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-console-config\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:28:15.394656 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.394655    5639 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62de3501-f48f-4f6b-a660-9df422b21bbf-console-oauth-config\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:28:15.394865 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.394670    5639 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-oauth-serving-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:28:15.394865 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.394685    5639 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62de3501-f48f-4f6b-a660-9df422b21bbf-console-serving-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:28:15.394865 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.394699    5639 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62de3501-f48f-4f6b-a660-9df422b21bbf-trusted-ca-bundle\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:28:15.394865 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.394713    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spxhg\" (UniqueName: \"kubernetes.io/projected/62de3501-f48f-4f6b-a660-9df422b21bbf-kube-api-access-spxhg\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:28:15.860486 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.860452    5639 generic.go:356] "Generic (PLEG): container finished" podID="62de3501-f48f-4f6b-a660-9df422b21bbf" containerID="73799674a1eec4df5c7cbd38b0868787bb18a63072a7db317fd543b6fd75c137" exitCode=0
Nov 02 01:28:15.860686 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.860500    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cdff6c64f-m9flm" event={"ID":"62de3501-f48f-4f6b-a660-9df422b21bbf","Type":"ContainerDied","Data":"73799674a1eec4df5c7cbd38b0868787bb18a63072a7db317fd543b6fd75c137"}
Nov 02 01:28:15.860686 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.860520    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cdff6c64f-m9flm"
Nov 02 01:28:15.860686 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.860565    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cdff6c64f-m9flm" event={"ID":"62de3501-f48f-4f6b-a660-9df422b21bbf","Type":"ContainerDied","Data":"038c49c5ced4963014a118738594943c668db5a881b00cc579a5a7530ba296f3"}
Nov 02 01:28:15.860686 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.860586    5639 scope.go:122] "RemoveContainer" containerID="73799674a1eec4df5c7cbd38b0868787bb18a63072a7db317fd543b6fd75c137"
Nov 02 01:28:15.869328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.869312    5639 scope.go:122] "RemoveContainer" containerID="73799674a1eec4df5c7cbd38b0868787bb18a63072a7db317fd543b6fd75c137"
Nov 02 01:28:15.869610 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:28:15.869590    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73799674a1eec4df5c7cbd38b0868787bb18a63072a7db317fd543b6fd75c137\": container with ID starting with 73799674a1eec4df5c7cbd38b0868787bb18a63072a7db317fd543b6fd75c137 not found: ID does not exist" containerID="73799674a1eec4df5c7cbd38b0868787bb18a63072a7db317fd543b6fd75c137"
Nov 02 01:28:15.869657 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:15.869621    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73799674a1eec4df5c7cbd38b0868787bb18a63072a7db317fd543b6fd75c137"} err="failed to get container status \"73799674a1eec4df5c7cbd38b0868787bb18a63072a7db317fd543b6fd75c137\": rpc error: code = NotFound desc = could not find container \"73799674a1eec4df5c7cbd38b0868787bb18a63072a7db317fd543b6fd75c137\": container with ID starting with 73799674a1eec4df5c7cbd38b0868787bb18a63072a7db317fd543b6fd75c137 not found: ID does not exist"
Nov 02 01:28:16.024159 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:16.024121    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cdff6c64f-m9flm"]
Nov 02 01:28:16.073958 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:16.073925    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cdff6c64f-m9flm"]
Nov 02 01:28:17.215476 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:28:17.215447    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62de3501-f48f-4f6b-a660-9df422b21bbf" path="/var/lib/kubelet/pods/62de3501-f48f-4f6b-a660-9df422b21bbf/volumes"
Nov 02 01:28:27.419773 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:28:27.419734    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:28:57.419087 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:28:57.418999    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:29:27.420007 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:29:27.419965    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:29:57.419097 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:29:57.419061    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:30:11.819628 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:11.819591    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548"]
Nov 02 01:30:11.820101 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:11.819910    5639 cpu_manager.go:414] "containerMap: removing container" podUID="62de3501-f48f-4f6b-a660-9df422b21bbf" containerName="console"
Nov 02 01:30:11.820101 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:11.819921    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62de3501-f48f-4f6b-a660-9df422b21bbf" containerName="console"
Nov 02 01:30:11.820101 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:11.819988    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="62de3501-f48f-4f6b-a660-9df422b21bbf" containerName="console"
Nov 02 01:30:11.822961 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:11.822943    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548"
Nov 02 01:30:11.877495 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:11.877462    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 02 01:30:11.877657 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:11.877559    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 02 01:30:11.877657 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:11.877593    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-z5hsq\""
Nov 02 01:30:11.892812 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:11.892779    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548"]
Nov 02 01:30:11.924905 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:11.924872    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5gz4\" (UniqueName: \"kubernetes.io/projected/0a5c4176-b82c-4a14-8690-c6af55cd6896-kube-api-access-l5gz4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548\" (UID: \"0a5c4176-b82c-4a14-8690-c6af55cd6896\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548"
Nov 02 01:30:11.925087 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:11.924938    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a5c4176-b82c-4a14-8690-c6af55cd6896-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548\" (UID: \"0a5c4176-b82c-4a14-8690-c6af55cd6896\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548"
Nov 02 01:30:11.925087 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:11.925063    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a5c4176-b82c-4a14-8690-c6af55cd6896-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548\" (UID: \"0a5c4176-b82c-4a14-8690-c6af55cd6896\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548"
Nov 02 01:30:12.006683 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.006653    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq"]
Nov 02 01:30:12.009962 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.009943    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq"
Nov 02 01:30:12.025433 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.025401    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a5c4176-b82c-4a14-8690-c6af55cd6896-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548\" (UID: \"0a5c4176-b82c-4a14-8690-c6af55cd6896\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548"
Nov 02 01:30:12.025562 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.025463    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5gz4\" (UniqueName: \"kubernetes.io/projected/0a5c4176-b82c-4a14-8690-c6af55cd6896-kube-api-access-l5gz4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548\" (UID: \"0a5c4176-b82c-4a14-8690-c6af55cd6896\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548"
Nov 02 01:30:12.025562 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.025507    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a5c4176-b82c-4a14-8690-c6af55cd6896-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548\" (UID: \"0a5c4176-b82c-4a14-8690-c6af55cd6896\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548"
Nov 02 01:30:12.025761 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.025742    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a5c4176-b82c-4a14-8690-c6af55cd6896-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548\" (UID: \"0a5c4176-b82c-4a14-8690-c6af55cd6896\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548"
Nov 02 01:30:12.025817 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.025800    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a5c4176-b82c-4a14-8690-c6af55cd6896-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548\" (UID: \"0a5c4176-b82c-4a14-8690-c6af55cd6896\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548"
Nov 02 01:30:12.062549 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.062517    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-5wn92\""
Nov 02 01:30:12.070788 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.070713    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq"]
Nov 02 01:30:12.090959 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.090931    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5gz4\" (UniqueName: \"kubernetes.io/projected/0a5c4176-b82c-4a14-8690-c6af55cd6896-kube-api-access-l5gz4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548\" (UID: \"0a5c4176-b82c-4a14-8690-c6af55cd6896\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548"
Nov 02 01:30:12.126101 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.126060    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32ac93ff-8819-4102-af81-3d6fb180d14b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq\" (UID: \"32ac93ff-8819-4102-af81-3d6fb180d14b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq"
Nov 02 01:30:12.126327 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.126118    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32ac93ff-8819-4102-af81-3d6fb180d14b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq\" (UID: \"32ac93ff-8819-4102-af81-3d6fb180d14b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq"
Nov 02 01:30:12.126327 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.126224    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd94x\" (UniqueName: \"kubernetes.io/projected/32ac93ff-8819-4102-af81-3d6fb180d14b-kube-api-access-cd94x\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq\" (UID: \"32ac93ff-8819-4102-af81-3d6fb180d14b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq"
Nov 02 01:30:12.139008 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.138967    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548"
Nov 02 01:30:12.227005 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.226972    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32ac93ff-8819-4102-af81-3d6fb180d14b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq\" (UID: \"32ac93ff-8819-4102-af81-3d6fb180d14b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq"
Nov 02 01:30:12.227143 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.227024    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd94x\" (UniqueName: \"kubernetes.io/projected/32ac93ff-8819-4102-af81-3d6fb180d14b-kube-api-access-cd94x\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq\" (UID: \"32ac93ff-8819-4102-af81-3d6fb180d14b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq"
Nov 02 01:30:12.227143 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.227055    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32ac93ff-8819-4102-af81-3d6fb180d14b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq\" (UID: \"32ac93ff-8819-4102-af81-3d6fb180d14b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq"
Nov 02 01:30:12.227383 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.227364    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32ac93ff-8819-4102-af81-3d6fb180d14b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq\" (UID: \"32ac93ff-8819-4102-af81-3d6fb180d14b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq"
Nov 02 01:30:12.227437 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.227394    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32ac93ff-8819-4102-af81-3d6fb180d14b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq\" (UID: \"32ac93ff-8819-4102-af81-3d6fb180d14b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq"
Nov 02 01:30:12.287735 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.287702    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd94x\" (UniqueName: \"kubernetes.io/projected/32ac93ff-8819-4102-af81-3d6fb180d14b-kube-api-access-cd94x\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq\" (UID: \"32ac93ff-8819-4102-af81-3d6fb180d14b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq"
Nov 02 01:30:12.325518 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.325442    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq"
Nov 02 01:30:12.432383 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.432353    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"]
Nov 02 01:30:12.435821 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.435801    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:30:12.493999 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.493963    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6ghtm\""
Nov 02 01:30:12.497387 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.497321    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 01:30:12.497545 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.497503    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 01:30:12.497545 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.497508    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 01:30:12.497955 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.497931    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 01:30:12.498128 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.498109    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 01:30:12.498377 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.498354    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 01:30:12.499266 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.499235    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"]
Nov 02 01:30:12.530041 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.530012    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-d5wdp\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:30:12.530182 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.530048    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-d5wdp\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:30:12.530182 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.530073    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p66wq\" (UniqueName: \"kubernetes.io/projected/e61d4236-18bc-4142-b11d-4691f0cd03ee-kube-api-access-p66wq\") pod \"istiod-openshift-gateway-7876bfcb74-d5wdp\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:30:12.530182 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.530149    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61d4236-18bc-4142-b11d-4691f0cd03ee-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-d5wdp\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:30:12.530388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.530216    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-d5wdp\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:30:12.530388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.530257    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-d5wdp\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:30:12.530388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.530356    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e61d4236-18bc-4142-b11d-4691f0cd03ee-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-d5wdp\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:30:12.559389 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.559361    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548"]
Nov 02 01:30:12.562798 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:30:12.562771    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5c4176_b82c_4a14_8690_c6af55cd6896.slice/crio-b322920767f9a47219b196087958ff52b433cdfbb68f455fae3fbf28f41f8db6 WatchSource:0}: Error finding container b322920767f9a47219b196087958ff52b433cdfbb68f455fae3fbf28f41f8db6: Status 404 returned error can't find the container with id b322920767f9a47219b196087958ff52b433cdfbb68f455fae3fbf28f41f8db6
Nov 02 01:30:12.631187 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.631150    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-d5wdp\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:30:12.631187 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.631185    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-d5wdp\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:30:12.631472 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.631210    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p66wq\" (UniqueName: \"kubernetes.io/projected/e61d4236-18bc-4142-b11d-4691f0cd03ee-kube-api-access-p66wq\") pod \"istiod-openshift-gateway-7876bfcb74-d5wdp\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:30:12.631472 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.631317    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61d4236-18bc-4142-b11d-4691f0cd03ee-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-d5wdp\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:30:12.631472 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.631359    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-d5wdp\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:30:12.631472 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.631384    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-d5wdp\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:30:12.631472 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.631430    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e61d4236-18bc-4142-b11d-4691f0cd03ee-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-d5wdp\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:30:12.632365 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.632246    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-d5wdp\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:30:12.633808 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.633782    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61d4236-18bc-4142-b11d-4691f0cd03ee-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-d5wdp\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:30:12.633906 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.633860    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-d5wdp\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:30:12.633997 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.633973    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-d5wdp\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:30:12.633997 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.633991    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e61d4236-18bc-4142-b11d-4691f0cd03ee-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-d5wdp\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:30:12.692831 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.692795    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-d5wdp\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:30:12.692972 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.692939    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p66wq\" (UniqueName: \"kubernetes.io/projected/e61d4236-18bc-4142-b11d-4691f0cd03ee-kube-api-access-p66wq\") pod \"istiod-openshift-gateway-7876bfcb74-d5wdp\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:30:12.700320 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.700295    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq"]
Nov 02 01:30:12.703954 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:30:12.703930    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ac93ff_8819_4102_af81_3d6fb180d14b.slice/crio-c157080b33b444004ef85fda7742bb3e0cabab4ea99533e8a256ae679dcfc717 WatchSource:0}: Error finding container c157080b33b444004ef85fda7742bb3e0cabab4ea99533e8a256ae679dcfc717: Status 404 returned error can't find the container with id c157080b33b444004ef85fda7742bb3e0cabab4ea99533e8a256ae679dcfc717
Nov 02 01:30:12.747559 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:12.747536    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:30:13.168313 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:13.168268    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"]
Nov 02 01:30:13.171062 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:30:13.171029    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode61d4236_18bc_4142_b11d_4691f0cd03ee.slice/crio-606c60f42b1ced9f50b9114ea6b473a99788a3168d53d93e7224cc82f00c6f88 WatchSource:0}: Error finding container 606c60f42b1ced9f50b9114ea6b473a99788a3168d53d93e7224cc82f00c6f88: Status 404 returned error can't find the container with id 606c60f42b1ced9f50b9114ea6b473a99788a3168d53d93e7224cc82f00c6f88
Nov 02 01:30:13.276091 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:13.276053    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq" event={"ID":"32ac93ff-8819-4102-af81-3d6fb180d14b","Type":"ContainerStarted","Data":"c157080b33b444004ef85fda7742bb3e0cabab4ea99533e8a256ae679dcfc717"}
Nov 02 01:30:13.277267 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:13.277244    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp" event={"ID":"e61d4236-18bc-4142-b11d-4691f0cd03ee","Type":"ContainerStarted","Data":"606c60f42b1ced9f50b9114ea6b473a99788a3168d53d93e7224cc82f00c6f88"}
Nov 02 01:30:13.278501 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:13.278471    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548" event={"ID":"0a5c4176-b82c-4a14-8690-c6af55cd6896","Type":"ContainerStarted","Data":"b322920767f9a47219b196087958ff52b433cdfbb68f455fae3fbf28f41f8db6"}
Nov 02 01:30:17.977713 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:17.977671    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:30:17.978080 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:17.977760    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:30:18.305141 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:18.305039    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp" event={"ID":"e61d4236-18bc-4142-b11d-4691f0cd03ee","Type":"ContainerStarted","Data":"8a63f8d1e1c31e762d1079e0468bc61c5fbab32730c062b533bc325d83a7b920"}
Nov 02 01:30:18.305361 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:18.305341    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:30:18.475119 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:18.475042    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp" podStartSLOduration=1.671814058 podStartE2EDuration="6.475021908s" totalImagesPullingTime="4.80320785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:18.474621497 +0000 UTC m=+283.378303612" watchObservedRunningTime="2026-11-02 01:30:18.475021908 +0000 UTC m=+283.378704016"
Nov 02 01:30:19.312714 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:19.312677    5639 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-d5wdp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:30:19.313162 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:19.312738    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp" podUID="e61d4236-18bc-4142-b11d-4691f0cd03ee" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:30:21.319062 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:21.318970    5639 generic.go:356] "Generic (PLEG): container finished" podID="32ac93ff-8819-4102-af81-3d6fb180d14b" containerID="2f7aced7cb16dcee4f9d9cbc6710762283cf54a9e670f065e70d88f61411c36b" exitCode=0
Nov 02 01:30:21.319508 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:21.319056    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq" event={"ID":"32ac93ff-8819-4102-af81-3d6fb180d14b","Type":"ContainerDied","Data":"2f7aced7cb16dcee4f9d9cbc6710762283cf54a9e670f065e70d88f61411c36b"}
Nov 02 01:30:21.320739 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:21.320700    5639 generic.go:356] "Generic (PLEG): container finished" podID="0a5c4176-b82c-4a14-8690-c6af55cd6896" containerID="60065682359feb05066c7e8cc884c99e93e0818f93b3fadcfe2d3006fb8da2e9" exitCode=0
Nov 02 01:30:21.320824 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:21.320758    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548" event={"ID":"0a5c4176-b82c-4a14-8690-c6af55cd6896","Type":"ContainerDied","Data":"60065682359feb05066c7e8cc884c99e93e0818f93b3fadcfe2d3006fb8da2e9"}
Nov 02 01:30:22.311415 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:22.311386    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:30:22.326890 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:22.326863    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq" event={"ID":"32ac93ff-8819-4102-af81-3d6fb180d14b","Type":"ContainerStarted","Data":"54dd730651900e6bcf0bb359f38e4752f83f3ebc96ad31744f01c7ab95ea7437"}
Nov 02 01:30:23.333891 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:23.333849    5639 generic.go:356] "Generic (PLEG): container finished" podID="32ac93ff-8819-4102-af81-3d6fb180d14b" containerID="54dd730651900e6bcf0bb359f38e4752f83f3ebc96ad31744f01c7ab95ea7437" exitCode=0
Nov 02 01:30:23.334373 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:23.333904    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq" event={"ID":"32ac93ff-8819-4102-af81-3d6fb180d14b","Type":"ContainerDied","Data":"54dd730651900e6bcf0bb359f38e4752f83f3ebc96ad31744f01c7ab95ea7437"}
Nov 02 01:30:25.349125 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:25.349092    5639 generic.go:356] "Generic (PLEG): container finished" podID="0a5c4176-b82c-4a14-8690-c6af55cd6896" containerID="10264f4aef06528d198abf01a78b96001141788b3e459bf9de87170b0adecd15" exitCode=0
Nov 02 01:30:25.349583 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:25.349171    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548" event={"ID":"0a5c4176-b82c-4a14-8690-c6af55cd6896","Type":"ContainerDied","Data":"10264f4aef06528d198abf01a78b96001141788b3e459bf9de87170b0adecd15"}
Nov 02 01:30:27.428043 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:30:27.428001    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:30:29.369705 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:29.369670    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq" event={"ID":"32ac93ff-8819-4102-af81-3d6fb180d14b","Type":"ContainerStarted","Data":"69f9bc8e32d16d9cc43554a3d74afa4fad0509412aeeb6e807b472f7b1bc9768"}
Nov 02 01:30:29.520247 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:29.520182    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq" podStartSLOduration=3.034471574 podStartE2EDuration="18.52016712s" totalImagesPullingTime="15.485695546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:29.519563334 +0000 UTC m=+294.423245450" watchObservedRunningTime="2026-11-02 01:30:29.52016712 +0000 UTC m=+294.423849234"
Nov 02 01:30:30.376724 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:30.376684    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548" event={"ID":"0a5c4176-b82c-4a14-8690-c6af55cd6896","Type":"ContainerStarted","Data":"4ba5230c830700ff7e1002ba2e3e0805e20b8343e8dfd0da4a1eaf98b18ac8a9"}
Nov 02 01:30:30.378936 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:30.378911    5639 generic.go:356] "Generic (PLEG): container finished" podID="32ac93ff-8819-4102-af81-3d6fb180d14b" containerID="69f9bc8e32d16d9cc43554a3d74afa4fad0509412aeeb6e807b472f7b1bc9768" exitCode=0
Nov 02 01:30:30.379034 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:30.378966    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq" event={"ID":"32ac93ff-8819-4102-af81-3d6fb180d14b","Type":"ContainerDied","Data":"69f9bc8e32d16d9cc43554a3d74afa4fad0509412aeeb6e807b472f7b1bc9768"}
Nov 02 01:30:30.495480 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:30.495425    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548" podStartSLOduration=3.701979319 podStartE2EDuration="19.495411352s" totalImagesPullingTime="15.793432033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:30.494699802 +0000 UTC m=+295.398381917" watchObservedRunningTime="2026-11-02 01:30:30.495411352 +0000 UTC m=+295.399093463"
Nov 02 01:30:31.384500 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:31.384468    5639 generic.go:356] "Generic (PLEG): container finished" podID="0a5c4176-b82c-4a14-8690-c6af55cd6896" containerID="4ba5230c830700ff7e1002ba2e3e0805e20b8343e8dfd0da4a1eaf98b18ac8a9" exitCode=0
Nov 02 01:30:31.385006 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:31.384555    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548" event={"ID":"0a5c4176-b82c-4a14-8690-c6af55cd6896","Type":"ContainerDied","Data":"4ba5230c830700ff7e1002ba2e3e0805e20b8343e8dfd0da4a1eaf98b18ac8a9"}
Nov 02 01:30:31.509395 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:31.509367    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq"
Nov 02 01:30:31.617648 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:31.617612    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ac93ff-8819-4102-af81-3d6fb180d14b-util\" (UniqueName: \"kubernetes.io/empty-dir/32ac93ff-8819-4102-af81-3d6fb180d14b-util\") pod \"32ac93ff-8819-4102-af81-3d6fb180d14b\" (UID: \"32ac93ff-8819-4102-af81-3d6fb180d14b\") "
Nov 02 01:30:31.617816 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:31.617678    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ac93ff-8819-4102-af81-3d6fb180d14b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/32ac93ff-8819-4102-af81-3d6fb180d14b-bundle\") pod \"32ac93ff-8819-4102-af81-3d6fb180d14b\" (UID: \"32ac93ff-8819-4102-af81-3d6fb180d14b\") "
Nov 02 01:30:31.617816 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:31.617723    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ac93ff-8819-4102-af81-3d6fb180d14b-kube-api-access-cd94x\" (UniqueName: \"kubernetes.io/projected/32ac93ff-8819-4102-af81-3d6fb180d14b-kube-api-access-cd94x\") pod \"32ac93ff-8819-4102-af81-3d6fb180d14b\" (UID: \"32ac93ff-8819-4102-af81-3d6fb180d14b\") "
Nov 02 01:30:31.618319 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:31.618251    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ac93ff-8819-4102-af81-3d6fb180d14b-bundle" pod "32ac93ff-8819-4102-af81-3d6fb180d14b" (UID: "32ac93ff-8819-4102-af81-3d6fb180d14b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:30:31.620108 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:31.620087    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ac93ff-8819-4102-af81-3d6fb180d14b-kube-api-access-cd94x" pod "32ac93ff-8819-4102-af81-3d6fb180d14b" (UID: "32ac93ff-8819-4102-af81-3d6fb180d14b"). InnerVolumeSpecName "kube-api-access-cd94x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:30:31.621991 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:31.621967    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ac93ff-8819-4102-af81-3d6fb180d14b-util" pod "32ac93ff-8819-4102-af81-3d6fb180d14b" (UID: "32ac93ff-8819-4102-af81-3d6fb180d14b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:30:31.718735 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:31.718639    5639 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32ac93ff-8819-4102-af81-3d6fb180d14b-util\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:31.718735 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:31.718673    5639 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32ac93ff-8819-4102-af81-3d6fb180d14b-bundle\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:31.718735 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:31.718687    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd94x\" (UniqueName: \"kubernetes.io/projected/32ac93ff-8819-4102-af81-3d6fb180d14b-kube-api-access-cd94x\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:32.390548 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:32.390509    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq" event={"ID":"32ac93ff-8819-4102-af81-3d6fb180d14b","Type":"ContainerDied","Data":"c157080b33b444004ef85fda7742bb3e0cabab4ea99533e8a256ae679dcfc717"}
Nov 02 01:30:32.390548 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:32.390542    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tq9kq"
Nov 02 01:30:32.391105 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:32.390561    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c157080b33b444004ef85fda7742bb3e0cabab4ea99533e8a256ae679dcfc717"
Nov 02 01:30:32.519460 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:32.519436    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548"
Nov 02 01:30:32.626421 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:32.626387    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a5c4176-b82c-4a14-8690-c6af55cd6896-kube-api-access-l5gz4\" (UniqueName: \"kubernetes.io/projected/0a5c4176-b82c-4a14-8690-c6af55cd6896-kube-api-access-l5gz4\") pod \"0a5c4176-b82c-4a14-8690-c6af55cd6896\" (UID: \"0a5c4176-b82c-4a14-8690-c6af55cd6896\") "
Nov 02 01:30:32.626584 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:32.626449    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a5c4176-b82c-4a14-8690-c6af55cd6896-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a5c4176-b82c-4a14-8690-c6af55cd6896-bundle\") pod \"0a5c4176-b82c-4a14-8690-c6af55cd6896\" (UID: \"0a5c4176-b82c-4a14-8690-c6af55cd6896\") "
Nov 02 01:30:32.626584 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:32.626490    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a5c4176-b82c-4a14-8690-c6af55cd6896-util\" (UniqueName: \"kubernetes.io/empty-dir/0a5c4176-b82c-4a14-8690-c6af55cd6896-util\") pod \"0a5c4176-b82c-4a14-8690-c6af55cd6896\" (UID: \"0a5c4176-b82c-4a14-8690-c6af55cd6896\") "
Nov 02 01:30:32.626878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:32.626855    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5c4176-b82c-4a14-8690-c6af55cd6896-bundle" pod "0a5c4176-b82c-4a14-8690-c6af55cd6896" (UID: "0a5c4176-b82c-4a14-8690-c6af55cd6896"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:30:32.628683 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:32.628661    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5c4176-b82c-4a14-8690-c6af55cd6896-kube-api-access-l5gz4" pod "0a5c4176-b82c-4a14-8690-c6af55cd6896" (UID: "0a5c4176-b82c-4a14-8690-c6af55cd6896"). InnerVolumeSpecName "kube-api-access-l5gz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:30:32.630501 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:32.630475    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5c4176-b82c-4a14-8690-c6af55cd6896-util" pod "0a5c4176-b82c-4a14-8690-c6af55cd6896" (UID: "0a5c4176-b82c-4a14-8690-c6af55cd6896"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:30:32.727342 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:32.727250    5639 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a5c4176-b82c-4a14-8690-c6af55cd6896-util\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:32.727342 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:32.727295    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5gz4\" (UniqueName: \"kubernetes.io/projected/0a5c4176-b82c-4a14-8690-c6af55cd6896-kube-api-access-l5gz4\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:32.727342 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:32.727307    5639 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a5c4176-b82c-4a14-8690-c6af55cd6896-bundle\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:33.396353 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:33.396318    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548" event={"ID":"0a5c4176-b82c-4a14-8690-c6af55cd6896","Type":"ContainerDied","Data":"b322920767f9a47219b196087958ff52b433cdfbb68f455fae3fbf28f41f8db6"}
Nov 02 01:30:33.396766 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:33.396366    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b322920767f9a47219b196087958ff52b433cdfbb68f455fae3fbf28f41f8db6"
Nov 02 01:30:33.396766 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:33.396418    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64488548"
Nov 02 01:30:52.404068 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.404024    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc"]
Nov 02 01:30:52.404733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.404543    5639 cpu_manager.go:414] "containerMap: removing container" podUID="32ac93ff-8819-4102-af81-3d6fb180d14b" containerName="util"
Nov 02 01:30:52.404733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.404564    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ac93ff-8819-4102-af81-3d6fb180d14b" containerName="util"
Nov 02 01:30:52.404733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.404591    5639 cpu_manager.go:414] "containerMap: removing container" podUID="0a5c4176-b82c-4a14-8690-c6af55cd6896" containerName="pull"
Nov 02 01:30:52.404733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.404600    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a5c4176-b82c-4a14-8690-c6af55cd6896" containerName="pull"
Nov 02 01:30:52.404733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.404618    5639 cpu_manager.go:414] "containerMap: removing container" podUID="0a5c4176-b82c-4a14-8690-c6af55cd6896" containerName="util"
Nov 02 01:30:52.404733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.404625    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a5c4176-b82c-4a14-8690-c6af55cd6896" containerName="util"
Nov 02 01:30:52.404733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.404639    5639 cpu_manager.go:414] "containerMap: removing container" podUID="32ac93ff-8819-4102-af81-3d6fb180d14b" containerName="pull"
Nov 02 01:30:52.404733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.404646    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ac93ff-8819-4102-af81-3d6fb180d14b" containerName="pull"
Nov 02 01:30:52.404733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.404656    5639 cpu_manager.go:414] "containerMap: removing container" podUID="0a5c4176-b82c-4a14-8690-c6af55cd6896" containerName="extract"
Nov 02 01:30:52.404733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.404664    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a5c4176-b82c-4a14-8690-c6af55cd6896" containerName="extract"
Nov 02 01:30:52.404733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.404681    5639 cpu_manager.go:414] "containerMap: removing container" podUID="32ac93ff-8819-4102-af81-3d6fb180d14b" containerName="extract"
Nov 02 01:30:52.404733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.404688    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ac93ff-8819-4102-af81-3d6fb180d14b" containerName="extract"
Nov 02 01:30:52.405328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.404777    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a5c4176-b82c-4a14-8690-c6af55cd6896" containerName="extract"
Nov 02 01:30:52.405328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.404790    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="32ac93ff-8819-4102-af81-3d6fb180d14b" containerName="extract"
Nov 02 01:30:52.406827 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.406806    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc"
Nov 02 01:30:52.501095 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.501056    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab6c4975-be5a-4419-b1b1-8031332d1203-tmp\") pod \"opendatahub-operator-controller-manager-5989ddd979-w57nc\" (UID: \"ab6c4975-be5a-4419-b1b1-8031332d1203\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc"
Nov 02 01:30:52.501441 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.501406    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ab6c4975-be5a-4419-b1b1-8031332d1203-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-w57nc\" (UID: \"ab6c4975-be5a-4419-b1b1-8031332d1203\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc"
Nov 02 01:30:52.501597 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.501575    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pk76\" (UniqueName: \"kubernetes.io/projected/ab6c4975-be5a-4419-b1b1-8031332d1203-kube-api-access-7pk76\") pod \"opendatahub-operator-controller-manager-5989ddd979-w57nc\" (UID: \"ab6c4975-be5a-4419-b1b1-8031332d1203\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc"
Nov 02 01:30:52.501669 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.501625    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ab6c4975-be5a-4419-b1b1-8031332d1203-webhook-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-w57nc\" (UID: \"ab6c4975-be5a-4419-b1b1-8031332d1203\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc"
Nov 02 01:30:52.501739 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.501708    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ab6c4975-be5a-4419-b1b1-8031332d1203-metrics-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-w57nc\" (UID: \"ab6c4975-be5a-4419-b1b1-8031332d1203\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc"
Nov 02 01:30:52.501957 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.501930    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ab6c4975-be5a-4419-b1b1-8031332d1203-manifests\") pod \"opendatahub-operator-controller-manager-5989ddd979-w57nc\" (UID: \"ab6c4975-be5a-4419-b1b1-8031332d1203\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc"
Nov 02 01:30:52.510828 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.510802    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc"]
Nov 02 01:30:52.534351 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.534319    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 01:30:52.534351 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.534340    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 01:30:52.534571 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.534394    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 01:30:52.534571 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.534438    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 01:30:52.534571 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.534438    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 01:30:52.534571 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.534515    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-vxbpb\""
Nov 02 01:30:52.603172 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.603143    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pk76\" (UniqueName: \"kubernetes.io/projected/ab6c4975-be5a-4419-b1b1-8031332d1203-kube-api-access-7pk76\") pod \"opendatahub-operator-controller-manager-5989ddd979-w57nc\" (UID: \"ab6c4975-be5a-4419-b1b1-8031332d1203\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc"
Nov 02 01:30:52.603392 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.603191    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ab6c4975-be5a-4419-b1b1-8031332d1203-webhook-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-w57nc\" (UID: \"ab6c4975-be5a-4419-b1b1-8031332d1203\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc"
Nov 02 01:30:52.603392 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.603231    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ab6c4975-be5a-4419-b1b1-8031332d1203-metrics-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-w57nc\" (UID: \"ab6c4975-be5a-4419-b1b1-8031332d1203\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc"
Nov 02 01:30:52.603392 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.603315    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ab6c4975-be5a-4419-b1b1-8031332d1203-manifests\") pod \"opendatahub-operator-controller-manager-5989ddd979-w57nc\" (UID: \"ab6c4975-be5a-4419-b1b1-8031332d1203\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc"
Nov 02 01:30:52.603392 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.603357    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab6c4975-be5a-4419-b1b1-8031332d1203-tmp\") pod \"opendatahub-operator-controller-manager-5989ddd979-w57nc\" (UID: \"ab6c4975-be5a-4419-b1b1-8031332d1203\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc"
Nov 02 01:30:52.603392 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.603379    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ab6c4975-be5a-4419-b1b1-8031332d1203-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-w57nc\" (UID: \"ab6c4975-be5a-4419-b1b1-8031332d1203\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc"
Nov 02 01:30:52.603827 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.603796    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ab6c4975-be5a-4419-b1b1-8031332d1203-manifests\") pod \"opendatahub-operator-controller-manager-5989ddd979-w57nc\" (UID: \"ab6c4975-be5a-4419-b1b1-8031332d1203\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc"
Nov 02 01:30:52.605850 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.605819    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab6c4975-be5a-4419-b1b1-8031332d1203-tmp\") pod \"opendatahub-operator-controller-manager-5989ddd979-w57nc\" (UID: \"ab6c4975-be5a-4419-b1b1-8031332d1203\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc"
Nov 02 01:30:52.606068 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.606046    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ab6c4975-be5a-4419-b1b1-8031332d1203-webhook-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-w57nc\" (UID: \"ab6c4975-be5a-4419-b1b1-8031332d1203\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc"
Nov 02 01:30:52.606114 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.606070    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ab6c4975-be5a-4419-b1b1-8031332d1203-metrics-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-w57nc\" (UID: \"ab6c4975-be5a-4419-b1b1-8031332d1203\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc"
Nov 02 01:30:52.606114 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.606083    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ab6c4975-be5a-4419-b1b1-8031332d1203-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-w57nc\" (UID: \"ab6c4975-be5a-4419-b1b1-8031332d1203\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc"
Nov 02 01:30:52.665054 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.664979    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pk76\" (UniqueName: \"kubernetes.io/projected/ab6c4975-be5a-4419-b1b1-8031332d1203-kube-api-access-7pk76\") pod \"opendatahub-operator-controller-manager-5989ddd979-w57nc\" (UID: \"ab6c4975-be5a-4419-b1b1-8031332d1203\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc"
Nov 02 01:30:52.720999 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:52.720966    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc"
Nov 02 01:30:53.216737 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:53.216712    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc"]
Nov 02 01:30:53.217939 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:30:53.217914    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab6c4975_be5a_4419_b1b1_8031332d1203.slice/crio-7cea410b2c8b60f6e627bf2d2ece73a3b6818a96d22ad42b3722e1c18367aac2 WatchSource:0}: Error finding container 7cea410b2c8b60f6e627bf2d2ece73a3b6818a96d22ad42b3722e1c18367aac2: Status 404 returned error can't find the container with id 7cea410b2c8b60f6e627bf2d2ece73a3b6818a96d22ad42b3722e1c18367aac2
Nov 02 01:30:53.471511 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:53.471426    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc" event={"ID":"ab6c4975-be5a-4419-b1b1-8031332d1203","Type":"ContainerStarted","Data":"7cea410b2c8b60f6e627bf2d2ece73a3b6818a96d22ad42b3722e1c18367aac2"}
Nov 02 01:30:57.419630 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:30:57.419589    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:30:57.490871 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:57.490840    5639 generic.go:356] "Generic (PLEG): container finished" podID="ab6c4975-be5a-4419-b1b1-8031332d1203" containerID="19af434345d5a1aebe5eb1887514038a9330aa35dc1d2fe1ca50e294d57c7a51" exitCode=0
Nov 02 01:30:57.491061 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:57.490920    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc" event={"ID":"ab6c4975-be5a-4419-b1b1-8031332d1203","Type":"ContainerDied","Data":"19af434345d5a1aebe5eb1887514038a9330aa35dc1d2fe1ca50e294d57c7a51"}
Nov 02 01:30:58.497352 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:58.497311    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc" event={"ID":"ab6c4975-be5a-4419-b1b1-8031332d1203","Type":"ContainerStarted","Data":"c74d89181cafe33622061816a698b47e71b09086839f0a9d0396b955474a8411"}
Nov 02 01:30:58.497754 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:58.497433    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc"
Nov 02 01:30:58.650295 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:30:58.650202    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc" podStartSLOduration=1.871671964 podStartE2EDuration="6.650184455s" totalImagesPullingTime="4.268968594s" totalInitContainerRuntime="509.543897ms" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:58.649688339 +0000 UTC m=+323.553370492" watchObservedRunningTime="2026-11-02 01:30:58.650184455 +0000 UTC m=+323.553866570"
Nov 02 01:31:09.503723 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:09.503690    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-w57nc"
Nov 02 01:31:27.419235 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:31:27.419195    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:31:38.854152 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:38.854118    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-79f447648c-xvbbq"]
Nov 02 01:31:38.859832 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:38.859803    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79f447648c-xvbbq"
Nov 02 01:31:38.962857 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:38.962819    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79f447648c-xvbbq"]
Nov 02 01:31:39.011416 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:39.011379    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61991930-092d-41ce-855b-6ed5041e460a-tmp\") pod \"kube-auth-proxy-79f447648c-xvbbq\" (UID: \"61991930-092d-41ce-855b-6ed5041e460a\") " pod="openshift-ingress/kube-auth-proxy-79f447648c-xvbbq"
Nov 02 01:31:39.011613 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:39.011445    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61991930-092d-41ce-855b-6ed5041e460a-tls-certs\") pod \"kube-auth-proxy-79f447648c-xvbbq\" (UID: \"61991930-092d-41ce-855b-6ed5041e460a\") " pod="openshift-ingress/kube-auth-proxy-79f447648c-xvbbq"
Nov 02 01:31:39.011613 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:39.011525    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 02 01:31:39.011613 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:39.011588    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 02 01:31:39.011613 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:39.011597    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjp4n\" (UniqueName: \"kubernetes.io/projected/61991930-092d-41ce-855b-6ed5041e460a-kube-api-access-rjp4n\") pod \"kube-auth-proxy-79f447648c-xvbbq\" (UID: \"61991930-092d-41ce-855b-6ed5041e460a\") " pod="openshift-ingress/kube-auth-proxy-79f447648c-xvbbq"
Nov 02 01:31:39.011794 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:39.011651    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-qv7s4\""
Nov 02 01:31:39.112696 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:39.112614    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61991930-092d-41ce-855b-6ed5041e460a-tmp\") pod \"kube-auth-proxy-79f447648c-xvbbq\" (UID: \"61991930-092d-41ce-855b-6ed5041e460a\") " pod="openshift-ingress/kube-auth-proxy-79f447648c-xvbbq"
Nov 02 01:31:39.112696 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:39.112661    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61991930-092d-41ce-855b-6ed5041e460a-tls-certs\") pod \"kube-auth-proxy-79f447648c-xvbbq\" (UID: \"61991930-092d-41ce-855b-6ed5041e460a\") " pod="openshift-ingress/kube-auth-proxy-79f447648c-xvbbq"
Nov 02 01:31:39.112924 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:39.112699    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjp4n\" (UniqueName: \"kubernetes.io/projected/61991930-092d-41ce-855b-6ed5041e460a-kube-api-access-rjp4n\") pod \"kube-auth-proxy-79f447648c-xvbbq\" (UID: \"61991930-092d-41ce-855b-6ed5041e460a\") " pod="openshift-ingress/kube-auth-proxy-79f447648c-xvbbq"
Nov 02 01:31:39.115032 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:39.115004    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61991930-092d-41ce-855b-6ed5041e460a-tmp\") pod \"kube-auth-proxy-79f447648c-xvbbq\" (UID: \"61991930-092d-41ce-855b-6ed5041e460a\") " pod="openshift-ingress/kube-auth-proxy-79f447648c-xvbbq"
Nov 02 01:31:39.115243 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:39.115228    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61991930-092d-41ce-855b-6ed5041e460a-tls-certs\") pod \"kube-auth-proxy-79f447648c-xvbbq\" (UID: \"61991930-092d-41ce-855b-6ed5041e460a\") " pod="openshift-ingress/kube-auth-proxy-79f447648c-xvbbq"
Nov 02 01:31:39.177186 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:39.177150    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjp4n\" (UniqueName: \"kubernetes.io/projected/61991930-092d-41ce-855b-6ed5041e460a-kube-api-access-rjp4n\") pod \"kube-auth-proxy-79f447648c-xvbbq\" (UID: \"61991930-092d-41ce-855b-6ed5041e460a\") " pod="openshift-ingress/kube-auth-proxy-79f447648c-xvbbq"
Nov 02 01:31:39.470392 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:39.470311    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79f447648c-xvbbq"
Nov 02 01:31:39.843520 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:39.843491    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79f447648c-xvbbq"]
Nov 02 01:31:39.846501 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:31:39.846474    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61991930_092d_41ce_855b_6ed5041e460a.slice/crio-e538a4a6b583639b5f9eed402d4f13b500c8f7069734920fca0597d9e06a4a78 WatchSource:0}: Error finding container e538a4a6b583639b5f9eed402d4f13b500c8f7069734920fca0597d9e06a4a78: Status 404 returned error can't find the container with id e538a4a6b583639b5f9eed402d4f13b500c8f7069734920fca0597d9e06a4a78
Nov 02 01:31:39.848446 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:39.848429    5639 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:31:40.668443 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:40.668408    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79f447648c-xvbbq" event={"ID":"61991930-092d-41ce-855b-6ed5041e460a","Type":"ContainerStarted","Data":"e538a4a6b583639b5f9eed402d4f13b500c8f7069734920fca0597d9e06a4a78"}
Nov 02 01:31:42.679217 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:42.679177    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79f447648c-xvbbq" event={"ID":"61991930-092d-41ce-855b-6ed5041e460a","Type":"ContainerStarted","Data":"f1f4e64c68e7bddd988f7ab31de5982f9f125effd9e966bc7a64281d91fa52d6"}
Nov 02 01:31:42.811905 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:42.811578    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-79f447648c-xvbbq" podStartSLOduration=2.153830239 podStartE2EDuration="4.811560899s" totalImagesPullingTime="2.65773066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:42.810383118 +0000 UTC m=+367.714065234" watchObservedRunningTime="2026-11-02 01:31:42.811560899 +0000 UTC m=+367.715243006"
Nov 02 01:31:43.610529 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:43.610496    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2m5qs"]
Nov 02 01:31:43.612645 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:43.612630    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2m5qs"
Nov 02 01:31:43.667727 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:43.667691    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 01:31:43.667870 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:43.667746    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 01:31:43.717202 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:43.717175    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bqwsk\""
Nov 02 01:31:43.718368 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:43.718342    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2m5qs"]
Nov 02 01:31:43.764533 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:43.764500    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqq8q\" (UniqueName: \"kubernetes.io/projected/5d935b2c-b6e6-4ff3-a6dc-c7e2e3c8fcd0-kube-api-access-qqq8q\") pod \"authorino-57f7f8659f-2m5qs\" (UID: \"5d935b2c-b6e6-4ff3-a6dc-c7e2e3c8fcd0\") " pod="kuadrant-system/authorino-57f7f8659f-2m5qs"
Nov 02 01:31:43.865777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:43.865691    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqq8q\" (UniqueName: \"kubernetes.io/projected/5d935b2c-b6e6-4ff3-a6dc-c7e2e3c8fcd0-kube-api-access-qqq8q\") pod \"authorino-57f7f8659f-2m5qs\" (UID: \"5d935b2c-b6e6-4ff3-a6dc-c7e2e3c8fcd0\") " pod="kuadrant-system/authorino-57f7f8659f-2m5qs"
Nov 02 01:31:43.928828 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:43.928783    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqq8q\" (UniqueName: \"kubernetes.io/projected/5d935b2c-b6e6-4ff3-a6dc-c7e2e3c8fcd0-kube-api-access-qqq8q\") pod \"authorino-57f7f8659f-2m5qs\" (UID: \"5d935b2c-b6e6-4ff3-a6dc-c7e2e3c8fcd0\") " pod="kuadrant-system/authorino-57f7f8659f-2m5qs"
Nov 02 01:31:43.932334 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:43.932308    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2m5qs"
Nov 02 01:31:44.299630 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:44.299560    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2m5qs"]
Nov 02 01:31:44.302484 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:31:44.302456    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d935b2c_b6e6_4ff3_a6dc_c7e2e3c8fcd0.slice/crio-9af6e6a113e2ffa3c7ad0199299b0acefe7460332add44433fe0b9976b8bdb45 WatchSource:0}: Error finding container 9af6e6a113e2ffa3c7ad0199299b0acefe7460332add44433fe0b9976b8bdb45: Status 404 returned error can't find the container with id 9af6e6a113e2ffa3c7ad0199299b0acefe7460332add44433fe0b9976b8bdb45
Nov 02 01:31:44.688348 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:44.688308    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2m5qs" event={"ID":"5d935b2c-b6e6-4ff3-a6dc-c7e2e3c8fcd0","Type":"ContainerStarted","Data":"9af6e6a113e2ffa3c7ad0199299b0acefe7460332add44433fe0b9976b8bdb45"}
Nov 02 01:31:46.493011 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:46.492939    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nzbhb"]
Nov 02 01:31:46.495511 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:46.495488    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nzbhb"
Nov 02 01:31:46.499393 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:46.499369    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:31:46.502188 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:46.502166    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"
Nov 02 01:31:46.552108 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:46.552075    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nzbhb"]
Nov 02 01:31:46.554470 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:46.554448    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8jvk2\""
Nov 02 01:31:46.554609 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:46.554553    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 02 01:31:46.591842 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:46.591809    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fh62\" (UniqueName: \"kubernetes.io/projected/bcaeb72e-96fa-429e-b5b6-f3ff966c93b2-kube-api-access-2fh62\") pod \"authorino-56c6c9ddd5-nzbhb\" (UID: \"bcaeb72e-96fa-429e-b5b6-f3ff966c93b2\") " pod="kuadrant-system/authorino-56c6c9ddd5-nzbhb"
Nov 02 01:31:46.592016 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:46.591871    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bghhl\" (UniqueName: \"kubernetes.io/projected/ab8b43c4-5ac8-48bd-a379-35672a7b44ca-kube-api-access-bghhl\") pod \"limitador-limitador-7c9b87d574-bjrd8\" (UID: \"ab8b43c4-5ac8-48bd-a379-35672a7b44ca\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"
Nov 02 01:31:46.592016 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:46.591935    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ab8b43c4-5ac8-48bd-a379-35672a7b44ca-config-file\") pod \"limitador-limitador-7c9b87d574-bjrd8\" (UID: \"ab8b43c4-5ac8-48bd-a379-35672a7b44ca\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"
Nov 02 01:31:46.614776 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:46.614741    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:31:46.692823 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:46.692787    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fh62\" (UniqueName: \"kubernetes.io/projected/bcaeb72e-96fa-429e-b5b6-f3ff966c93b2-kube-api-access-2fh62\") pod \"authorino-56c6c9ddd5-nzbhb\" (UID: \"bcaeb72e-96fa-429e-b5b6-f3ff966c93b2\") " pod="kuadrant-system/authorino-56c6c9ddd5-nzbhb"
Nov 02 01:31:46.692998 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:46.692849    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bghhl\" (UniqueName: \"kubernetes.io/projected/ab8b43c4-5ac8-48bd-a379-35672a7b44ca-kube-api-access-bghhl\") pod \"limitador-limitador-7c9b87d574-bjrd8\" (UID: \"ab8b43c4-5ac8-48bd-a379-35672a7b44ca\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"
Nov 02 01:31:46.692998 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:46.692911    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ab8b43c4-5ac8-48bd-a379-35672a7b44ca-config-file\") pod \"limitador-limitador-7c9b87d574-bjrd8\" (UID: \"ab8b43c4-5ac8-48bd-a379-35672a7b44ca\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"
Nov 02 01:31:46.693786 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:46.693761    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ab8b43c4-5ac8-48bd-a379-35672a7b44ca-config-file\") pod \"limitador-limitador-7c9b87d574-bjrd8\" (UID: \"ab8b43c4-5ac8-48bd-a379-35672a7b44ca\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"
Nov 02 01:31:46.753780 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:46.753706    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fh62\" (UniqueName: \"kubernetes.io/projected/bcaeb72e-96fa-429e-b5b6-f3ff966c93b2-kube-api-access-2fh62\") pod \"authorino-56c6c9ddd5-nzbhb\" (UID: \"bcaeb72e-96fa-429e-b5b6-f3ff966c93b2\") " pod="kuadrant-system/authorino-56c6c9ddd5-nzbhb"
Nov 02 01:31:46.753780 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:46.753766    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bghhl\" (UniqueName: \"kubernetes.io/projected/ab8b43c4-5ac8-48bd-a379-35672a7b44ca-kube-api-access-bghhl\") pod \"limitador-limitador-7c9b87d574-bjrd8\" (UID: \"ab8b43c4-5ac8-48bd-a379-35672a7b44ca\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"
Nov 02 01:31:46.798778 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:46.798742    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:31:46.810756 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:46.810728    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nzbhb"
Nov 02 01:31:46.818896 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:46.818873    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"
Nov 02 01:31:47.198524 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:47.198491    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nzbhb"]
Nov 02 01:31:47.202459 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:31:47.202427    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbcaeb72e_96fa_429e_b5b6_f3ff966c93b2.slice/crio-799f1ddeb71911d57a7c33d1581cb5683de13ffc0d91114ec39b51aae46eaa65 WatchSource:0}: Error finding container 799f1ddeb71911d57a7c33d1581cb5683de13ffc0d91114ec39b51aae46eaa65: Status 404 returned error can't find the container with id 799f1ddeb71911d57a7c33d1581cb5683de13ffc0d91114ec39b51aae46eaa65
Nov 02 01:31:47.248973 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:47.248939    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:31:47.251741 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:31:47.251715    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab8b43c4_5ac8_48bd_a379_35672a7b44ca.slice/crio-e5f9663b72550bbc8a01ffdb960975777c029c381f82ecc5153f60044756f58b WatchSource:0}: Error finding container e5f9663b72550bbc8a01ffdb960975777c029c381f82ecc5153f60044756f58b: Status 404 returned error can't find the container with id e5f9663b72550bbc8a01ffdb960975777c029c381f82ecc5153f60044756f58b
Nov 02 01:31:47.704535 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:47.704497    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-bjrd8" event={"ID":"ab8b43c4-5ac8-48bd-a379-35672a7b44ca","Type":"ContainerStarted","Data":"e5f9663b72550bbc8a01ffdb960975777c029c381f82ecc5153f60044756f58b"}
Nov 02 01:31:47.705822 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:47.705795    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nzbhb" event={"ID":"bcaeb72e-96fa-429e-b5b6-f3ff966c93b2","Type":"ContainerStarted","Data":"799f1ddeb71911d57a7c33d1581cb5683de13ffc0d91114ec39b51aae46eaa65"}
Nov 02 01:31:48.712884 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:48.712830    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2m5qs" event={"ID":"5d935b2c-b6e6-4ff3-a6dc-c7e2e3c8fcd0","Type":"ContainerStarted","Data":"0d396eaecf12d037e87c9e6caab2be15f76993238d904a40b6618b64baf3d6fe"}
Nov 02 01:31:48.714718 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:48.714686    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nzbhb" event={"ID":"bcaeb72e-96fa-429e-b5b6-f3ff966c93b2","Type":"ContainerStarted","Data":"008e8da7919cb97da0dc29d1edcbadbfdabc1cc72123ff2d53f9d2af98cf465b"}
Nov 02 01:31:48.868779 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:48.868723    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-2m5qs" podStartSLOduration=2.114220846 podStartE2EDuration="5.868704988s" totalImagesPullingTime="3.754484142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:48.867087879 +0000 UTC m=+373.770770007" watchObservedRunningTime="2026-11-02 01:31:48.868704988 +0000 UTC m=+373.772387104"
Nov 02 01:31:48.993986 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:48.993925    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-nzbhb" podStartSLOduration=1.6979189689999998 podStartE2EDuration="2.993909425s" totalImagesPullingTime="1.295990456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:48.993330013 +0000 UTC m=+373.897012130" watchObservedRunningTime="2026-11-02 01:31:48.993909425 +0000 UTC m=+373.897591541"
Nov 02 01:31:49.034954 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:49.034914    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2m5qs"]
Nov 02 01:31:49.720971 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:49.720935    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-bjrd8" event={"ID":"ab8b43c4-5ac8-48bd-a379-35672a7b44ca","Type":"ContainerStarted","Data":"74596bf48b4f4a661e60481f29872d7bc4329a35e0b1116cba605a73a0480f06"}
Nov 02 01:31:49.721445 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:49.721137    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"
Nov 02 01:31:49.841344 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:49.841273    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-bjrd8" podStartSLOduration=2.233316623 podStartE2EDuration="3.841255737s" totalImagesPullingTime="1.607939114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:49.840234636 +0000 UTC m=+374.743916750" watchObservedRunningTime="2026-11-02 01:31:49.841255737 +0000 UTC m=+374.744937871"
Nov 02 01:31:50.725455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:50.725412    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-2m5qs" podUID="5d935b2c-b6e6-4ff3-a6dc-c7e2e3c8fcd0" containerName="authorino" containerID="cri-o://0d396eaecf12d037e87c9e6caab2be15f76993238d904a40b6618b64baf3d6fe" gracePeriod=30
Nov 02 01:31:50.968771 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:50.968747    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2m5qs"
Nov 02 01:31:51.041531 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:51.041436    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d935b2c-b6e6-4ff3-a6dc-c7e2e3c8fcd0-kube-api-access-qqq8q\" (UniqueName: \"kubernetes.io/projected/5d935b2c-b6e6-4ff3-a6dc-c7e2e3c8fcd0-kube-api-access-qqq8q\") pod \"5d935b2c-b6e6-4ff3-a6dc-c7e2e3c8fcd0\" (UID: \"5d935b2c-b6e6-4ff3-a6dc-c7e2e3c8fcd0\") "
Nov 02 01:31:51.043793 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:51.043759    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d935b2c-b6e6-4ff3-a6dc-c7e2e3c8fcd0-kube-api-access-qqq8q" pod "5d935b2c-b6e6-4ff3-a6dc-c7e2e3c8fcd0" (UID: "5d935b2c-b6e6-4ff3-a6dc-c7e2e3c8fcd0"). InnerVolumeSpecName "kube-api-access-qqq8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:31:51.143018 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:51.142977    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqq8q\" (UniqueName: \"kubernetes.io/projected/5d935b2c-b6e6-4ff3-a6dc-c7e2e3c8fcd0-kube-api-access-qqq8q\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:31:51.733246 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:51.733213    5639 generic.go:356] "Generic (PLEG): container finished" podID="5d935b2c-b6e6-4ff3-a6dc-c7e2e3c8fcd0" containerID="0d396eaecf12d037e87c9e6caab2be15f76993238d904a40b6618b64baf3d6fe" exitCode=0
Nov 02 01:31:51.733681 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:51.733264    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2m5qs"
Nov 02 01:31:51.733681 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:51.733306    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2m5qs" event={"ID":"5d935b2c-b6e6-4ff3-a6dc-c7e2e3c8fcd0","Type":"ContainerDied","Data":"0d396eaecf12d037e87c9e6caab2be15f76993238d904a40b6618b64baf3d6fe"}
Nov 02 01:31:51.733681 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:51.733347    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2m5qs" event={"ID":"5d935b2c-b6e6-4ff3-a6dc-c7e2e3c8fcd0","Type":"ContainerDied","Data":"9af6e6a113e2ffa3c7ad0199299b0acefe7460332add44433fe0b9976b8bdb45"}
Nov 02 01:31:51.733681 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:51.733367    5639 scope.go:122] "RemoveContainer" containerID="0d396eaecf12d037e87c9e6caab2be15f76993238d904a40b6618b64baf3d6fe"
Nov 02 01:31:51.741702 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:51.741672    5639 scope.go:122] "RemoveContainer" containerID="0d396eaecf12d037e87c9e6caab2be15f76993238d904a40b6618b64baf3d6fe"
Nov 02 01:31:51.741977 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:31:51.741959    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d396eaecf12d037e87c9e6caab2be15f76993238d904a40b6618b64baf3d6fe\": container with ID starting with 0d396eaecf12d037e87c9e6caab2be15f76993238d904a40b6618b64baf3d6fe not found: ID does not exist" containerID="0d396eaecf12d037e87c9e6caab2be15f76993238d904a40b6618b64baf3d6fe"
Nov 02 01:31:51.742047 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:51.741986    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d396eaecf12d037e87c9e6caab2be15f76993238d904a40b6618b64baf3d6fe"} err="failed to get container status \"0d396eaecf12d037e87c9e6caab2be15f76993238d904a40b6618b64baf3d6fe\": rpc error: code = NotFound desc = could not find container \"0d396eaecf12d037e87c9e6caab2be15f76993238d904a40b6618b64baf3d6fe\": container with ID starting with 0d396eaecf12d037e87c9e6caab2be15f76993238d904a40b6618b64baf3d6fe not found: ID does not exist"
Nov 02 01:31:51.894480 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:51.894444    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2m5qs"]
Nov 02 01:31:51.943900 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:51.943867    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2m5qs"]
Nov 02 01:31:53.216818 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:31:53.216775    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d935b2c-b6e6-4ff3-a6dc-c7e2e3c8fcd0" path="/var/lib/kubelet/pods/5d935b2c-b6e6-4ff3-a6dc-c7e2e3c8fcd0/volumes"
Nov 02 01:31:57.421761 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:31:57.421720    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:32:00.727004 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:00.726967    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"
Nov 02 01:32:25.734116 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:25.734080    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65b55dd8bb-jrkxd"]
Nov 02 01:32:25.734633 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:25.734523    5639 cpu_manager.go:414] "containerMap: removing container" podUID="5d935b2c-b6e6-4ff3-a6dc-c7e2e3c8fcd0" containerName="authorino"
Nov 02 01:32:25.734633 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:25.734539    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d935b2c-b6e6-4ff3-a6dc-c7e2e3c8fcd0" containerName="authorino"
Nov 02 01:32:25.734633 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:25.734607    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d935b2c-b6e6-4ff3-a6dc-c7e2e3c8fcd0" containerName="authorino"
Nov 02 01:32:25.736530 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:25.736514    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65b55dd8bb-jrkxd"
Nov 02 01:32:25.753365 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:25.753336    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d1816d-03d5-4be7-9b26-69c6da974752-metrics-certs\") pod \"maas-controller-65b55dd8bb-jrkxd\" (UID: \"71d1816d-03d5-4be7-9b26-69c6da974752\") " pod="opendatahub/maas-controller-65b55dd8bb-jrkxd"
Nov 02 01:32:25.753510 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:25.753422    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29z9w\" (UniqueName: \"kubernetes.io/projected/71d1816d-03d5-4be7-9b26-69c6da974752-kube-api-access-29z9w\") pod \"maas-controller-65b55dd8bb-jrkxd\" (UID: \"71d1816d-03d5-4be7-9b26-69c6da974752\") " pod="opendatahub/maas-controller-65b55dd8bb-jrkxd"
Nov 02 01:32:25.753510 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:25.753456    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71d1816d-03d5-4be7-9b26-69c6da974752-webhook-cert\") pod \"maas-controller-65b55dd8bb-jrkxd\" (UID: \"71d1816d-03d5-4be7-9b26-69c6da974752\") " pod="opendatahub/maas-controller-65b55dd8bb-jrkxd"
Nov 02 01:32:25.838381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:25.838342    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 01:32:25.838381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:25.838356    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 01:32:25.838381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:25.838346    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ghnpp\""
Nov 02 01:32:25.838621 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:25.838423    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 01:32:25.838621 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:25.838432    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 01:32:25.838621 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:25.838481    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 01:32:25.839512 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:25.839492    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65b55dd8bb-jrkxd"]
Nov 02 01:32:25.854208 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:25.854174    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d1816d-03d5-4be7-9b26-69c6da974752-metrics-certs\") pod \"maas-controller-65b55dd8bb-jrkxd\" (UID: \"71d1816d-03d5-4be7-9b26-69c6da974752\") " pod="opendatahub/maas-controller-65b55dd8bb-jrkxd"
Nov 02 01:32:25.854404 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:25.854264    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29z9w\" (UniqueName: \"kubernetes.io/projected/71d1816d-03d5-4be7-9b26-69c6da974752-kube-api-access-29z9w\") pod \"maas-controller-65b55dd8bb-jrkxd\" (UID: \"71d1816d-03d5-4be7-9b26-69c6da974752\") " pod="opendatahub/maas-controller-65b55dd8bb-jrkxd"
Nov 02 01:32:25.854404 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:25.854339    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71d1816d-03d5-4be7-9b26-69c6da974752-webhook-cert\") pod \"maas-controller-65b55dd8bb-jrkxd\" (UID: \"71d1816d-03d5-4be7-9b26-69c6da974752\") " pod="opendatahub/maas-controller-65b55dd8bb-jrkxd"
Nov 02 01:32:25.856826 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:25.856807    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71d1816d-03d5-4be7-9b26-69c6da974752-webhook-cert\") pod \"maas-controller-65b55dd8bb-jrkxd\" (UID: \"71d1816d-03d5-4be7-9b26-69c6da974752\") " pod="opendatahub/maas-controller-65b55dd8bb-jrkxd"
Nov 02 01:32:25.856938 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:25.856918    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d1816d-03d5-4be7-9b26-69c6da974752-metrics-certs\") pod \"maas-controller-65b55dd8bb-jrkxd\" (UID: \"71d1816d-03d5-4be7-9b26-69c6da974752\") " pod="opendatahub/maas-controller-65b55dd8bb-jrkxd"
Nov 02 01:32:25.915101 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:25.915077    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29z9w\" (UniqueName: \"kubernetes.io/projected/71d1816d-03d5-4be7-9b26-69c6da974752-kube-api-access-29z9w\") pod \"maas-controller-65b55dd8bb-jrkxd\" (UID: \"71d1816d-03d5-4be7-9b26-69c6da974752\") " pod="opendatahub/maas-controller-65b55dd8bb-jrkxd"
Nov 02 01:32:26.047931 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:26.047840    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65b55dd8bb-jrkxd"
Nov 02 01:32:26.467249 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:26.467220    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65b55dd8bb-jrkxd"]
Nov 02 01:32:26.470000 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:32:26.469979    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d1816d_03d5_4be7_9b26_69c6da974752.slice/crio-de14d6264f752819ec71bfa097ad3c0dd98974d37b06e5f693dd6da70196bbea WatchSource:0}: Error finding container de14d6264f752819ec71bfa097ad3c0dd98974d37b06e5f693dd6da70196bbea: Status 404 returned error can't find the container with id de14d6264f752819ec71bfa097ad3c0dd98974d37b06e5f693dd6da70196bbea
Nov 02 01:32:26.885438 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:26.885405    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65b55dd8bb-jrkxd" event={"ID":"71d1816d-03d5-4be7-9b26-69c6da974752","Type":"ContainerStarted","Data":"de14d6264f752819ec71bfa097ad3c0dd98974d37b06e5f693dd6da70196bbea"}
Nov 02 01:32:27.436548 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:32:27.436499    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:32:29.903623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:29.903591    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65b55dd8bb-jrkxd" event={"ID":"71d1816d-03d5-4be7-9b26-69c6da974752","Type":"ContainerStarted","Data":"d428e16296d41f7ee3edb8c00b14c911250646a5679d1d8375bf143f620d5beb"}
Nov 02 01:32:29.904007 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:29.903725    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65b55dd8bb-jrkxd"
Nov 02 01:32:30.022204 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:30.022145    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-65b55dd8bb-jrkxd" podStartSLOduration=2.002372538 podStartE2EDuration="5.022128827s" totalImagesPullingTime="3.019756289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:32:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:30.020915529 +0000 UTC m=+414.924597665" watchObservedRunningTime="2026-11-02 01:32:30.022128827 +0000 UTC m=+414.925810931"
Nov 02 01:32:40.910422 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:32:40.910392    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65b55dd8bb-jrkxd"
Nov 02 01:32:57.426628 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:32:57.426588    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:33:07.575327 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:07.575269    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"]
Nov 02 01:33:07.581937 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:07.581915    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 01:33:07.680032 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:07.679990    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"]
Nov 02 01:33:07.713112 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:07.713081    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qkrbz\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 01:33:07.713345 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:07.713122    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qkrbz\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 01:33:07.713345 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:07.713150    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qkrbz\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 01:33:07.713345 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:07.713295    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qkrbz\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 01:33:07.713345 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:07.713332    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh5l8\" (UniqueName: \"kubernetes.io/projected/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-kube-api-access-lh5l8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qkrbz\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 01:33:07.713534 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:07.713434    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qkrbz\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 01:33:07.713534 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:07.713463    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qkrbz\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 01:33:07.814406 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:07.814372    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qkrbz\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 01:33:07.814604 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:07.814413    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qkrbz\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 01:33:07.814604 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:07.814433    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qkrbz\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 01:33:07.814604 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:07.814452    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qkrbz\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 01:33:07.814604 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:07.814471    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh5l8\" (UniqueName: \"kubernetes.io/projected/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-kube-api-access-lh5l8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qkrbz\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 01:33:07.814604 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:07.814493    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qkrbz\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 01:33:07.814604 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:07.814510    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qkrbz\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 01:33:07.815357 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:07.815305    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qkrbz\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 01:33:07.816923 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:07.816899    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qkrbz\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 01:33:07.817004 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:07.816989    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qkrbz\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 01:33:07.817401 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:07.817383    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qkrbz\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 01:33:07.817512 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:07.817492    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qkrbz\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 01:33:07.875868 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:07.875840    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qkrbz\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 01:33:07.876052 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:07.876029    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh5l8\" (UniqueName: \"kubernetes.io/projected/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-kube-api-access-lh5l8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qkrbz\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 01:33:07.893732 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:07.893706    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 01:33:08.188763 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:08.188671    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:33:08.323936 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:08.323895    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"]
Nov 02 01:33:08.328071 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:33:08.328042    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604ec2ee_bfbc_4ba6_aa7f_0e61310b5563.slice/crio-864aa1eca13c2a35e8e4699dd4f1ecdd8553ed0846550450fcb6ff2b3bf71180 WatchSource:0}: Error finding container 864aa1eca13c2a35e8e4699dd4f1ecdd8553ed0846550450fcb6ff2b3bf71180: Status 404 returned error can't find the container with id 864aa1eca13c2a35e8e4699dd4f1ecdd8553ed0846550450fcb6ff2b3bf71180
Nov 02 01:33:08.330751 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:08.330718    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:33:08.330823 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:08.330784    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:33:09.070604 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:09.070571    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz" event={"ID":"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563","Type":"ContainerStarted","Data":"25a7d87c9b08321d64b35f27358c936d387c5407dff592e65f8444d2d8636db1"}
Nov 02 01:33:09.071081 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:09.070623    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz" event={"ID":"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563","Type":"ContainerStarted","Data":"864aa1eca13c2a35e8e4699dd4f1ecdd8553ed0846550450fcb6ff2b3bf71180"}
Nov 02 01:33:09.071081 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:09.070725    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 01:33:09.072326 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:09.072308    5639 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-qkrbz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:33:09.072417 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:09.072352    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz" podUID="604ec2ee-bfbc-4ba6-aa7f-0e61310b5563" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:33:09.191098 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:09.191037    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz" podStartSLOduration=2.191017392 podStartE2EDuration="2.191017392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:33:09.190233907 +0000 UTC m=+454.093916034" watchObservedRunningTime="2026-11-02 01:33:09.191017392 +0000 UTC m=+454.094699507"
Nov 02 01:33:10.076851 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.076824    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 01:33:10.299758 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.299725    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"]
Nov 02 01:33:10.299984 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.299957    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp" podUID="e61d4236-18bc-4142-b11d-4691f0cd03ee" containerName="discovery" containerID="cri-o://8a63f8d1e1c31e762d1079e0468bc61c5fbab32730c062b533bc325d83a7b920" gracePeriod=30
Nov 02 01:33:10.311057 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.311026    5639 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-d5wdp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.21:8080/ready\": dial tcp 10.134.0.21:8080: connect: connection refused" start-of-body=
Nov 02 01:33:10.311188 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.311094    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp" podUID="e61d4236-18bc-4142-b11d-4691f0cd03ee" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.21:8080/ready\": dial tcp 10.134.0.21:8080: connect: connection refused"
Nov 02 01:33:10.538515 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.538492    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:33:10.645323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.645267    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e61d4236-18bc-4142-b11d-4691f0cd03ee-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61d4236-18bc-4142-b11d-4691f0cd03ee-local-certs\") pod \"e61d4236-18bc-4142-b11d-4691f0cd03ee\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") "
Nov 02 01:33:10.645504 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.645355    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-kubeconfig\") pod \"e61d4236-18bc-4142-b11d-4691f0cd03ee\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") "
Nov 02 01:33:10.645504 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.645418    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-csr-dns-cert\") pod \"e61d4236-18bc-4142-b11d-4691f0cd03ee\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") "
Nov 02 01:33:10.645504 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.645463    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-csr-ca-configmap\") pod \"e61d4236-18bc-4142-b11d-4691f0cd03ee\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") "
Nov 02 01:33:10.645504 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.645490    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e61d4236-18bc-4142-b11d-4691f0cd03ee-cacerts\" (UniqueName: \"kubernetes.io/secret/e61d4236-18bc-4142-b11d-4691f0cd03ee-cacerts\") pod \"e61d4236-18bc-4142-b11d-4691f0cd03ee\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") "
Nov 02 01:33:10.645720 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.645535    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e61d4236-18bc-4142-b11d-4691f0cd03ee-kube-api-access-p66wq\" (UniqueName: \"kubernetes.io/projected/e61d4236-18bc-4142-b11d-4691f0cd03ee-kube-api-access-p66wq\") pod \"e61d4236-18bc-4142-b11d-4691f0cd03ee\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") "
Nov 02 01:33:10.645720 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.645589    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-token\" (UniqueName: \"kubernetes.io/projected/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-token\") pod \"e61d4236-18bc-4142-b11d-4691f0cd03ee\" (UID: \"e61d4236-18bc-4142-b11d-4691f0cd03ee\") "
Nov 02 01:33:10.645915 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.645860    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-csr-ca-configmap" pod "e61d4236-18bc-4142-b11d-4691f0cd03ee" (UID: "e61d4236-18bc-4142-b11d-4691f0cd03ee"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:33:10.646141 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.645993    5639 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-csr-ca-configmap\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:10.648266 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.648161    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-kubeconfig" pod "e61d4236-18bc-4142-b11d-4691f0cd03ee" (UID: "e61d4236-18bc-4142-b11d-4691f0cd03ee"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:33:10.648266 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.648218    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-csr-dns-cert" pod "e61d4236-18bc-4142-b11d-4691f0cd03ee" (UID: "e61d4236-18bc-4142-b11d-4691f0cd03ee"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:33:10.648424 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.648322    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61d4236-18bc-4142-b11d-4691f0cd03ee-local-certs" pod "e61d4236-18bc-4142-b11d-4691f0cd03ee" (UID: "e61d4236-18bc-4142-b11d-4691f0cd03ee"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:33:10.648424 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.648350    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61d4236-18bc-4142-b11d-4691f0cd03ee-kube-api-access-p66wq" pod "e61d4236-18bc-4142-b11d-4691f0cd03ee" (UID: "e61d4236-18bc-4142-b11d-4691f0cd03ee"). InnerVolumeSpecName "kube-api-access-p66wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:33:10.648504 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.648491    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e61d4236-18bc-4142-b11d-4691f0cd03ee-cacerts" pod "e61d4236-18bc-4142-b11d-4691f0cd03ee" (UID: "e61d4236-18bc-4142-b11d-4691f0cd03ee"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:33:10.648644 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.648629    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-token" pod "e61d4236-18bc-4142-b11d-4691f0cd03ee" (UID: "e61d4236-18bc-4142-b11d-4691f0cd03ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:33:10.746736 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.746640    5639 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61d4236-18bc-4142-b11d-4691f0cd03ee-local-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:10.746736 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.746670    5639 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-kubeconfig\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:10.746736 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.746683    5639 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-csr-dns-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:10.746736 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.746697    5639 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e61d4236-18bc-4142-b11d-4691f0cd03ee-cacerts\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:10.746736 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.746711    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p66wq\" (UniqueName: \"kubernetes.io/projected/e61d4236-18bc-4142-b11d-4691f0cd03ee-kube-api-access-p66wq\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:10.746736 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:10.746723    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61d4236-18bc-4142-b11d-4691f0cd03ee-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:11.088356 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:11.088320    5639 generic.go:356] "Generic (PLEG): container finished" podID="e61d4236-18bc-4142-b11d-4691f0cd03ee" containerID="8a63f8d1e1c31e762d1079e0468bc61c5fbab32730c062b533bc325d83a7b920" exitCode=0
Nov 02 01:33:11.088829 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:11.088386    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"
Nov 02 01:33:11.088829 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:11.088402    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp" event={"ID":"e61d4236-18bc-4142-b11d-4691f0cd03ee","Type":"ContainerDied","Data":"8a63f8d1e1c31e762d1079e0468bc61c5fbab32730c062b533bc325d83a7b920"}
Nov 02 01:33:11.088829 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:11.088479    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp" event={"ID":"e61d4236-18bc-4142-b11d-4691f0cd03ee","Type":"ContainerDied","Data":"606c60f42b1ced9f50b9114ea6b473a99788a3168d53d93e7224cc82f00c6f88"}
Nov 02 01:33:11.088829 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:11.088499    5639 scope.go:122] "RemoveContainer" containerID="8a63f8d1e1c31e762d1079e0468bc61c5fbab32730c062b533bc325d83a7b920"
Nov 02 01:33:11.097303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:11.097264    5639 scope.go:122] "RemoveContainer" containerID="8a63f8d1e1c31e762d1079e0468bc61c5fbab32730c062b533bc325d83a7b920"
Nov 02 01:33:11.097583 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:33:11.097564    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a63f8d1e1c31e762d1079e0468bc61c5fbab32730c062b533bc325d83a7b920\": container with ID starting with 8a63f8d1e1c31e762d1079e0468bc61c5fbab32730c062b533bc325d83a7b920 not found: ID does not exist" containerID="8a63f8d1e1c31e762d1079e0468bc61c5fbab32730c062b533bc325d83a7b920"
Nov 02 01:33:11.097638 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:11.097592    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a63f8d1e1c31e762d1079e0468bc61c5fbab32730c062b533bc325d83a7b920"} err="failed to get container status \"8a63f8d1e1c31e762d1079e0468bc61c5fbab32730c062b533bc325d83a7b920\": rpc error: code = NotFound desc = could not find container \"8a63f8d1e1c31e762d1079e0468bc61c5fbab32730c062b533bc325d83a7b920\": container with ID starting with 8a63f8d1e1c31e762d1079e0468bc61c5fbab32730c062b533bc325d83a7b920 not found: ID does not exist"
Nov 02 01:33:11.252503 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:11.252467    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"]
Nov 02 01:33:11.302340 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:11.302311    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-d5wdp"]
Nov 02 01:33:12.246351 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:12.246313    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-kvd2d"]
Nov 02 01:33:12.246841 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:12.246675    5639 cpu_manager.go:414] "containerMap: removing container" podUID="e61d4236-18bc-4142-b11d-4691f0cd03ee" containerName="discovery"
Nov 02 01:33:12.246841 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:12.246686    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e61d4236-18bc-4142-b11d-4691f0cd03ee" containerName="discovery"
Nov 02 01:33:12.246841 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:12.246773    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="e61d4236-18bc-4142-b11d-4691f0cd03ee" containerName="discovery"
Nov 02 01:33:12.251088 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:12.251064    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-kvd2d"
Nov 02 01:33:12.306205 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:12.306177    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-kvd2d"]
Nov 02 01:33:12.362382 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:12.362346    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78tmm\" (UniqueName: \"kubernetes.io/projected/ffd1bb54-c677-409c-93bf-6b5c244fa9ac-kube-api-access-78tmm\") pod \"authorino-6f6944bcdd-kvd2d\" (UID: \"ffd1bb54-c677-409c-93bf-6b5c244fa9ac\") " pod="kuadrant-system/authorino-6f6944bcdd-kvd2d"
Nov 02 01:33:12.463196 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:12.463142    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78tmm\" (UniqueName: \"kubernetes.io/projected/ffd1bb54-c677-409c-93bf-6b5c244fa9ac-kube-api-access-78tmm\") pod \"authorino-6f6944bcdd-kvd2d\" (UID: \"ffd1bb54-c677-409c-93bf-6b5c244fa9ac\") " pod="kuadrant-system/authorino-6f6944bcdd-kvd2d"
Nov 02 01:33:12.525287 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:12.525211    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78tmm\" (UniqueName: \"kubernetes.io/projected/ffd1bb54-c677-409c-93bf-6b5c244fa9ac-kube-api-access-78tmm\") pod \"authorino-6f6944bcdd-kvd2d\" (UID: \"ffd1bb54-c677-409c-93bf-6b5c244fa9ac\") " pod="kuadrant-system/authorino-6f6944bcdd-kvd2d"
Nov 02 01:33:12.563783 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:12.563747    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-kvd2d"
Nov 02 01:33:12.611065 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:12.611034    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-kvd2d"]
Nov 02 01:33:12.640587 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:12.640561    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b777d9d5-sp4z2"]
Nov 02 01:33:12.645022 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:12.645002    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b777d9d5-sp4z2"
Nov 02 01:33:12.701458 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:12.701424    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b777d9d5-sp4z2"]
Nov 02 01:33:12.765101 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:12.765070    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdcmp\" (UniqueName: \"kubernetes.io/projected/e7a4933f-3f9c-4a30-80df-0a3d408fd0d6-kube-api-access-vdcmp\") pod \"authorino-5b777d9d5-sp4z2\" (UID: \"e7a4933f-3f9c-4a30-80df-0a3d408fd0d6\") " pod="kuadrant-system/authorino-5b777d9d5-sp4z2"
Nov 02 01:33:12.865906 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:12.865871    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdcmp\" (UniqueName: \"kubernetes.io/projected/e7a4933f-3f9c-4a30-80df-0a3d408fd0d6-kube-api-access-vdcmp\") pod \"authorino-5b777d9d5-sp4z2\" (UID: \"e7a4933f-3f9c-4a30-80df-0a3d408fd0d6\") " pod="kuadrant-system/authorino-5b777d9d5-sp4z2"
Nov 02 01:33:12.927174 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:12.927134    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdcmp\" (UniqueName: \"kubernetes.io/projected/e7a4933f-3f9c-4a30-80df-0a3d408fd0d6-kube-api-access-vdcmp\") pod \"authorino-5b777d9d5-sp4z2\" (UID: \"e7a4933f-3f9c-4a30-80df-0a3d408fd0d6\") " pod="kuadrant-system/authorino-5b777d9d5-sp4z2"
Nov 02 01:33:12.933961 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:12.933929    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-kvd2d"]
Nov 02 01:33:12.937505 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:33:12.937480    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffd1bb54_c677_409c_93bf_6b5c244fa9ac.slice/crio-462a7d9ade2f8772ad3f705b717942a63cf73d7ac56a0fbc06ee851ad82287e0 WatchSource:0}: Error finding container 462a7d9ade2f8772ad3f705b717942a63cf73d7ac56a0fbc06ee851ad82287e0: Status 404 returned error can't find the container with id 462a7d9ade2f8772ad3f705b717942a63cf73d7ac56a0fbc06ee851ad82287e0
Nov 02 01:33:12.957426 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:12.957401    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b777d9d5-sp4z2"
Nov 02 01:33:12.977630 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:12.977602    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b777d9d5-sp4z2"]
Nov 02 01:33:13.004525 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.004498    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687b947b6c-w5459"]
Nov 02 01:33:13.008978 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.008957    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687b947b6c-w5459"
Nov 02 01:33:13.064812 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.064786    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687b947b6c-w5459"]
Nov 02 01:33:13.065771 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.065749    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 02 01:33:13.099348 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.099308    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-kvd2d" event={"ID":"ffd1bb54-c677-409c-93bf-6b5c244fa9ac","Type":"ContainerStarted","Data":"462a7d9ade2f8772ad3f705b717942a63cf73d7ac56a0fbc06ee851ad82287e0"}
Nov 02 01:33:13.102954 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.102930    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-4wp6c"]
Nov 02 01:33:13.106374 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.106352    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-4wp6c"
Nov 02 01:33:13.165070 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.165000    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-4wp6c"]
Nov 02 01:33:13.165995 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.165969    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 02 01:33:13.165995 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.165968    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-xgdhx\""
Nov 02 01:33:13.169389 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.169363    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb62a2e9-fc33-49ed-a2b9-9d46c8305edf-tls-cert\") pod \"authorino-687b947b6c-w5459\" (UID: \"cb62a2e9-fc33-49ed-a2b9-9d46c8305edf\") " pod="kuadrant-system/authorino-687b947b6c-w5459"
Nov 02 01:33:13.169505 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.169398    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k496c\" (UniqueName: \"kubernetes.io/projected/cb62a2e9-fc33-49ed-a2b9-9d46c8305edf-kube-api-access-k496c\") pod \"authorino-687b947b6c-w5459\" (UID: \"cb62a2e9-fc33-49ed-a2b9-9d46c8305edf\") " pod="kuadrant-system/authorino-687b947b6c-w5459"
Nov 02 01:33:13.217951 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.217919    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e61d4236-18bc-4142-b11d-4691f0cd03ee" path="/var/lib/kubelet/pods/e61d4236-18bc-4142-b11d-4691f0cd03ee/volumes"
Nov 02 01:33:13.270142 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.270111    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psvw6\" (UniqueName: \"kubernetes.io/projected/b072a543-fc27-4360-bbf8-b3d41fcb468d-kube-api-access-psvw6\") pod \"kserve-controller-manager-776cc66d47-4wp6c\" (UID: \"b072a543-fc27-4360-bbf8-b3d41fcb468d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4wp6c"
Nov 02 01:33:13.270596 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.270174    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb62a2e9-fc33-49ed-a2b9-9d46c8305edf-tls-cert\") pod \"authorino-687b947b6c-w5459\" (UID: \"cb62a2e9-fc33-49ed-a2b9-9d46c8305edf\") " pod="kuadrant-system/authorino-687b947b6c-w5459"
Nov 02 01:33:13.270596 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.270193    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k496c\" (UniqueName: \"kubernetes.io/projected/cb62a2e9-fc33-49ed-a2b9-9d46c8305edf-kube-api-access-k496c\") pod \"authorino-687b947b6c-w5459\" (UID: \"cb62a2e9-fc33-49ed-a2b9-9d46c8305edf\") " pod="kuadrant-system/authorino-687b947b6c-w5459"
Nov 02 01:33:13.270596 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.270315    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b072a543-fc27-4360-bbf8-b3d41fcb468d-cert\") pod \"kserve-controller-manager-776cc66d47-4wp6c\" (UID: \"b072a543-fc27-4360-bbf8-b3d41fcb468d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4wp6c"
Nov 02 01:33:13.272692 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.272675    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb62a2e9-fc33-49ed-a2b9-9d46c8305edf-tls-cert\") pod \"authorino-687b947b6c-w5459\" (UID: \"cb62a2e9-fc33-49ed-a2b9-9d46c8305edf\") " pod="kuadrant-system/authorino-687b947b6c-w5459"
Nov 02 01:33:13.328890 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.328802    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b777d9d5-sp4z2"]
Nov 02 01:33:13.331788 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.331764    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k496c\" (UniqueName: \"kubernetes.io/projected/cb62a2e9-fc33-49ed-a2b9-9d46c8305edf-kube-api-access-k496c\") pod \"authorino-687b947b6c-w5459\" (UID: \"cb62a2e9-fc33-49ed-a2b9-9d46c8305edf\") " pod="kuadrant-system/authorino-687b947b6c-w5459"
Nov 02 01:33:13.332113 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:33:13.332088    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7a4933f_3f9c_4a30_80df_0a3d408fd0d6.slice/crio-5d457e728f571c72ea942a227419d837776d053f5d04cefb37cb2bd44b473507 WatchSource:0}: Error finding container 5d457e728f571c72ea942a227419d837776d053f5d04cefb37cb2bd44b473507: Status 404 returned error can't find the container with id 5d457e728f571c72ea942a227419d837776d053f5d04cefb37cb2bd44b473507
Nov 02 01:33:13.371664 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.371635    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psvw6\" (UniqueName: \"kubernetes.io/projected/b072a543-fc27-4360-bbf8-b3d41fcb468d-kube-api-access-psvw6\") pod \"kserve-controller-manager-776cc66d47-4wp6c\" (UID: \"b072a543-fc27-4360-bbf8-b3d41fcb468d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4wp6c"
Nov 02 01:33:13.371814 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.371722    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b072a543-fc27-4360-bbf8-b3d41fcb468d-cert\") pod \"kserve-controller-manager-776cc66d47-4wp6c\" (UID: \"b072a543-fc27-4360-bbf8-b3d41fcb468d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4wp6c"
Nov 02 01:33:13.374174 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.374151    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b072a543-fc27-4360-bbf8-b3d41fcb468d-cert\") pod \"kserve-controller-manager-776cc66d47-4wp6c\" (UID: \"b072a543-fc27-4360-bbf8-b3d41fcb468d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4wp6c"
Nov 02 01:33:13.433109 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.433041    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psvw6\" (UniqueName: \"kubernetes.io/projected/b072a543-fc27-4360-bbf8-b3d41fcb468d-kube-api-access-psvw6\") pod \"kserve-controller-manager-776cc66d47-4wp6c\" (UID: \"b072a543-fc27-4360-bbf8-b3d41fcb468d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4wp6c"
Nov 02 01:33:13.627760 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.627727    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687b947b6c-w5459"
Nov 02 01:33:13.720815 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.720741    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-4wp6c"
Nov 02 01:33:13.998805 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:13.998735    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687b947b6c-w5459"]
Nov 02 01:33:14.001707 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:33:14.001683    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb62a2e9_fc33_49ed_a2b9_9d46c8305edf.slice/crio-86d63186b5aa29ac1d6b4be6a103baf745148b823b82e3e7db0b96b54054e7ea WatchSource:0}: Error finding container 86d63186b5aa29ac1d6b4be6a103baf745148b823b82e3e7db0b96b54054e7ea: Status 404 returned error can't find the container with id 86d63186b5aa29ac1d6b4be6a103baf745148b823b82e3e7db0b96b54054e7ea
Nov 02 01:33:14.105185 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:14.105155    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b777d9d5-sp4z2" event={"ID":"e7a4933f-3f9c-4a30-80df-0a3d408fd0d6","Type":"ContainerStarted","Data":"5d457e728f571c72ea942a227419d837776d053f5d04cefb37cb2bd44b473507"}
Nov 02 01:33:14.106424 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:14.106397    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-687b947b6c-w5459" event={"ID":"cb62a2e9-fc33-49ed-a2b9-9d46c8305edf","Type":"ContainerStarted","Data":"86d63186b5aa29ac1d6b4be6a103baf745148b823b82e3e7db0b96b54054e7ea"}
Nov 02 01:33:14.110737 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:14.110714    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-4wp6c"]
Nov 02 01:33:14.113673 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:33:14.113649    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb072a543_fc27_4360_bbf8_b3d41fcb468d.slice/crio-c9bd460f3a4035472ba3a5bcd81d72b219b0b47560ce36339d76e3413df97b51 WatchSource:0}: Error finding container c9bd460f3a4035472ba3a5bcd81d72b219b0b47560ce36339d76e3413df97b51: Status 404 returned error can't find the container with id c9bd460f3a4035472ba3a5bcd81d72b219b0b47560ce36339d76e3413df97b51
Nov 02 01:33:15.132491 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:15.132454    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-kvd2d" event={"ID":"ffd1bb54-c677-409c-93bf-6b5c244fa9ac","Type":"ContainerStarted","Data":"0d5366b280493d393dfbfbefdbcfc131c91286da916917ef08058b0a57ca4ca2"}
Nov 02 01:33:15.132994 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:15.132549    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-kvd2d" podUID="ffd1bb54-c677-409c-93bf-6b5c244fa9ac" containerName="authorino" containerID="cri-o://0d5366b280493d393dfbfbefdbcfc131c91286da916917ef08058b0a57ca4ca2" gracePeriod=30
Nov 02 01:33:15.138170 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:15.138125    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-4wp6c" event={"ID":"b072a543-fc27-4360-bbf8-b3d41fcb468d","Type":"ContainerStarted","Data":"c9bd460f3a4035472ba3a5bcd81d72b219b0b47560ce36339d76e3413df97b51"}
Nov 02 01:33:15.139973 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:15.139944    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b777d9d5-sp4z2" event={"ID":"e7a4933f-3f9c-4a30-80df-0a3d408fd0d6","Type":"ContainerStarted","Data":"6c8e5949372a4b0ac767fa5ebb586dbc6c5156e9468e625121880fff19fa34e9"}
Nov 02 01:33:15.140102 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:15.139987    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5b777d9d5-sp4z2" podUID="e7a4933f-3f9c-4a30-80df-0a3d408fd0d6" containerName="authorino" containerID="cri-o://6c8e5949372a4b0ac767fa5ebb586dbc6c5156e9468e625121880fff19fa34e9" gracePeriod=30
Nov 02 01:33:15.387647 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:15.387563    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-kvd2d"
Nov 02 01:33:15.425644 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:15.425620    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b777d9d5-sp4z2"
Nov 02 01:33:15.435341 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:15.435232    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-kvd2d" podStartSLOduration=1.9957445649999999 podStartE2EDuration="3.435212075s" totalImagesPullingTime="1.43946751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:33:15.320270626 +0000 UTC m=+460.223952741" watchObservedRunningTime="2026-11-02 01:33:15.435212075 +0000 UTC m=+460.338894190"
Nov 02 01:33:15.488601 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:15.488566    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd1bb54-c677-409c-93bf-6b5c244fa9ac-kube-api-access-78tmm\" (UniqueName: \"kubernetes.io/projected/ffd1bb54-c677-409c-93bf-6b5c244fa9ac-kube-api-access-78tmm\") pod \"ffd1bb54-c677-409c-93bf-6b5c244fa9ac\" (UID: \"ffd1bb54-c677-409c-93bf-6b5c244fa9ac\") "
Nov 02 01:33:15.491213 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:15.491186    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd1bb54-c677-409c-93bf-6b5c244fa9ac-kube-api-access-78tmm" pod "ffd1bb54-c677-409c-93bf-6b5c244fa9ac" (UID: "ffd1bb54-c677-409c-93bf-6b5c244fa9ac"). InnerVolumeSpecName "kube-api-access-78tmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:33:15.589565 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:15.589531    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7a4933f-3f9c-4a30-80df-0a3d408fd0d6-kube-api-access-vdcmp\" (UniqueName: \"kubernetes.io/projected/e7a4933f-3f9c-4a30-80df-0a3d408fd0d6-kube-api-access-vdcmp\") pod \"e7a4933f-3f9c-4a30-80df-0a3d408fd0d6\" (UID: \"e7a4933f-3f9c-4a30-80df-0a3d408fd0d6\") "
Nov 02 01:33:15.589753 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:15.589738    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78tmm\" (UniqueName: \"kubernetes.io/projected/ffd1bb54-c677-409c-93bf-6b5c244fa9ac-kube-api-access-78tmm\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:15.592030 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:15.591993    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7a4933f-3f9c-4a30-80df-0a3d408fd0d6-kube-api-access-vdcmp" pod "e7a4933f-3f9c-4a30-80df-0a3d408fd0d6" (UID: "e7a4933f-3f9c-4a30-80df-0a3d408fd0d6"). InnerVolumeSpecName "kube-api-access-vdcmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:33:15.691208 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:15.691116    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdcmp\" (UniqueName: \"kubernetes.io/projected/e7a4933f-3f9c-4a30-80df-0a3d408fd0d6-kube-api-access-vdcmp\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:16.147575 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:16.147532    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-687b947b6c-w5459" event={"ID":"cb62a2e9-fc33-49ed-a2b9-9d46c8305edf","Type":"ContainerStarted","Data":"301c38c9108984c25e9c031e9f27baa0f5edc2269479b4c180c02e658a826b23"}
Nov 02 01:33:16.151660 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:16.151615    5639 generic.go:356] "Generic (PLEG): container finished" podID="ffd1bb54-c677-409c-93bf-6b5c244fa9ac" containerID="0d5366b280493d393dfbfbefdbcfc131c91286da916917ef08058b0a57ca4ca2" exitCode=0
Nov 02 01:33:16.151814 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:16.151730    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-kvd2d" event={"ID":"ffd1bb54-c677-409c-93bf-6b5c244fa9ac","Type":"ContainerDied","Data":"0d5366b280493d393dfbfbefdbcfc131c91286da916917ef08058b0a57ca4ca2"}
Nov 02 01:33:16.151814 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:16.151763    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-kvd2d" event={"ID":"ffd1bb54-c677-409c-93bf-6b5c244fa9ac","Type":"ContainerDied","Data":"462a7d9ade2f8772ad3f705b717942a63cf73d7ac56a0fbc06ee851ad82287e0"}
Nov 02 01:33:16.151814 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:16.151784    5639 scope.go:122] "RemoveContainer" containerID="0d5366b280493d393dfbfbefdbcfc131c91286da916917ef08058b0a57ca4ca2"
Nov 02 01:33:16.151982 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:16.151968    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-kvd2d"
Nov 02 01:33:16.165324 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:16.164341    5639 generic.go:356] "Generic (PLEG): container finished" podID="e7a4933f-3f9c-4a30-80df-0a3d408fd0d6" containerID="6c8e5949372a4b0ac767fa5ebb586dbc6c5156e9468e625121880fff19fa34e9" exitCode=0
Nov 02 01:33:16.165324 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:16.164436    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b777d9d5-sp4z2" event={"ID":"e7a4933f-3f9c-4a30-80df-0a3d408fd0d6","Type":"ContainerDied","Data":"6c8e5949372a4b0ac767fa5ebb586dbc6c5156e9468e625121880fff19fa34e9"}
Nov 02 01:33:16.165324 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:16.164468    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b777d9d5-sp4z2" event={"ID":"e7a4933f-3f9c-4a30-80df-0a3d408fd0d6","Type":"ContainerDied","Data":"5d457e728f571c72ea942a227419d837776d053f5d04cefb37cb2bd44b473507"}
Nov 02 01:33:16.165324 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:16.164532    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b777d9d5-sp4z2"
Nov 02 01:33:16.172869 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:16.172844    5639 scope.go:122] "RemoveContainer" containerID="0d5366b280493d393dfbfbefdbcfc131c91286da916917ef08058b0a57ca4ca2"
Nov 02 01:33:16.173201 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:33:16.173168    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d5366b280493d393dfbfbefdbcfc131c91286da916917ef08058b0a57ca4ca2\": container with ID starting with 0d5366b280493d393dfbfbefdbcfc131c91286da916917ef08058b0a57ca4ca2 not found: ID does not exist" containerID="0d5366b280493d393dfbfbefdbcfc131c91286da916917ef08058b0a57ca4ca2"
Nov 02 01:33:16.173307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:16.173213    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d5366b280493d393dfbfbefdbcfc131c91286da916917ef08058b0a57ca4ca2"} err="failed to get container status \"0d5366b280493d393dfbfbefdbcfc131c91286da916917ef08058b0a57ca4ca2\": rpc error: code = NotFound desc = could not find container \"0d5366b280493d393dfbfbefdbcfc131c91286da916917ef08058b0a57ca4ca2\": container with ID starting with 0d5366b280493d393dfbfbefdbcfc131c91286da916917ef08058b0a57ca4ca2 not found: ID does not exist"
Nov 02 01:33:16.173307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:16.173238    5639 scope.go:122] "RemoveContainer" containerID="6c8e5949372a4b0ac767fa5ebb586dbc6c5156e9468e625121880fff19fa34e9"
Nov 02 01:33:16.186372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:16.186351    5639 scope.go:122] "RemoveContainer" containerID="6c8e5949372a4b0ac767fa5ebb586dbc6c5156e9468e625121880fff19fa34e9"
Nov 02 01:33:16.187026 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:33:16.186999    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c8e5949372a4b0ac767fa5ebb586dbc6c5156e9468e625121880fff19fa34e9\": container with ID starting with 6c8e5949372a4b0ac767fa5ebb586dbc6c5156e9468e625121880fff19fa34e9 not found: ID does not exist" containerID="6c8e5949372a4b0ac767fa5ebb586dbc6c5156e9468e625121880fff19fa34e9"
Nov 02 01:33:16.187108 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:16.187038    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c8e5949372a4b0ac767fa5ebb586dbc6c5156e9468e625121880fff19fa34e9"} err="failed to get container status \"6c8e5949372a4b0ac767fa5ebb586dbc6c5156e9468e625121880fff19fa34e9\": rpc error: code = NotFound desc = could not find container \"6c8e5949372a4b0ac767fa5ebb586dbc6c5156e9468e625121880fff19fa34e9\": container with ID starting with 6c8e5949372a4b0ac767fa5ebb586dbc6c5156e9468e625121880fff19fa34e9 not found: ID does not exist"
Nov 02 01:33:16.264015 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:16.263949    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-687b947b6c-w5459" podStartSLOduration=3.070803381 podStartE2EDuration="4.263929685s" totalImagesPullingTime="1.193126304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:33:16.263160565 +0000 UTC m=+461.166842679" watchObservedRunningTime="2026-11-02 01:33:16.263929685 +0000 UTC m=+461.167611801"
Nov 02 01:33:16.291743 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:16.290858    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nzbhb"]
Nov 02 01:33:16.291743 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:16.291025    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-nzbhb" podUID="bcaeb72e-96fa-429e-b5b6-f3ff966c93b2" containerName="authorino" containerID="cri-o://008e8da7919cb97da0dc29d1edcbadbfdabc1cc72123ff2d53f9d2af98cf465b" gracePeriod=30
Nov 02 01:33:16.376405 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:16.376372    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b777d9d5-sp4z2"]
Nov 02 01:33:16.382885 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:16.382855    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5b777d9d5-sp4z2"]
Nov 02 01:33:16.493562 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:16.493530    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-kvd2d"]
Nov 02 01:33:16.500074 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:16.500013    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-kvd2d"]
Nov 02 01:33:16.563683 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:16.563657    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nzbhb"
Nov 02 01:33:16.704079 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:16.703988    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcaeb72e-96fa-429e-b5b6-f3ff966c93b2-kube-api-access-2fh62\" (UniqueName: \"kubernetes.io/projected/bcaeb72e-96fa-429e-b5b6-f3ff966c93b2-kube-api-access-2fh62\") pod \"bcaeb72e-96fa-429e-b5b6-f3ff966c93b2\" (UID: \"bcaeb72e-96fa-429e-b5b6-f3ff966c93b2\") "
Nov 02 01:33:16.706758 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:16.706725    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcaeb72e-96fa-429e-b5b6-f3ff966c93b2-kube-api-access-2fh62" pod "bcaeb72e-96fa-429e-b5b6-f3ff966c93b2" (UID: "bcaeb72e-96fa-429e-b5b6-f3ff966c93b2"). InnerVolumeSpecName "kube-api-access-2fh62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:33:16.804740 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:16.804709    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fh62\" (UniqueName: \"kubernetes.io/projected/bcaeb72e-96fa-429e-b5b6-f3ff966c93b2-kube-api-access-2fh62\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:17.170975 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:17.170939    5639 generic.go:356] "Generic (PLEG): container finished" podID="bcaeb72e-96fa-429e-b5b6-f3ff966c93b2" containerID="008e8da7919cb97da0dc29d1edcbadbfdabc1cc72123ff2d53f9d2af98cf465b" exitCode=0
Nov 02 01:33:17.171424 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:17.170999    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nzbhb"
Nov 02 01:33:17.171424 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:17.171025    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nzbhb" event={"ID":"bcaeb72e-96fa-429e-b5b6-f3ff966c93b2","Type":"ContainerDied","Data":"008e8da7919cb97da0dc29d1edcbadbfdabc1cc72123ff2d53f9d2af98cf465b"}
Nov 02 01:33:17.171424 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:17.171063    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nzbhb" event={"ID":"bcaeb72e-96fa-429e-b5b6-f3ff966c93b2","Type":"ContainerDied","Data":"799f1ddeb71911d57a7c33d1581cb5683de13ffc0d91114ec39b51aae46eaa65"}
Nov 02 01:33:17.171424 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:17.171079    5639 scope.go:122] "RemoveContainer" containerID="008e8da7919cb97da0dc29d1edcbadbfdabc1cc72123ff2d53f9d2af98cf465b"
Nov 02 01:33:17.173922 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:17.173887    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-4wp6c" event={"ID":"b072a543-fc27-4360-bbf8-b3d41fcb468d","Type":"ContainerStarted","Data":"e8bbf194ec5640f3fc44be9cf9adc96ae1e4cbce45e18d680d1e6ef3f5a7365b"}
Nov 02 01:33:17.180144 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:17.180128    5639 scope.go:122] "RemoveContainer" containerID="008e8da7919cb97da0dc29d1edcbadbfdabc1cc72123ff2d53f9d2af98cf465b"
Nov 02 01:33:17.180434 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:33:17.180414    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"008e8da7919cb97da0dc29d1edcbadbfdabc1cc72123ff2d53f9d2af98cf465b\": container with ID starting with 008e8da7919cb97da0dc29d1edcbadbfdabc1cc72123ff2d53f9d2af98cf465b not found: ID does not exist" containerID="008e8da7919cb97da0dc29d1edcbadbfdabc1cc72123ff2d53f9d2af98cf465b"
Nov 02 01:33:17.180501 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:17.180445    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"008e8da7919cb97da0dc29d1edcbadbfdabc1cc72123ff2d53f9d2af98cf465b"} err="failed to get container status \"008e8da7919cb97da0dc29d1edcbadbfdabc1cc72123ff2d53f9d2af98cf465b\": rpc error: code = NotFound desc = could not find container \"008e8da7919cb97da0dc29d1edcbadbfdabc1cc72123ff2d53f9d2af98cf465b\": container with ID starting with 008e8da7919cb97da0dc29d1edcbadbfdabc1cc72123ff2d53f9d2af98cf465b not found: ID does not exist"
Nov 02 01:33:17.217720 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:17.217651    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7a4933f-3f9c-4a30-80df-0a3d408fd0d6" path="/var/lib/kubelet/pods/e7a4933f-3f9c-4a30-80df-0a3d408fd0d6/volumes"
Nov 02 01:33:17.217961 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:17.217949    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffd1bb54-c677-409c-93bf-6b5c244fa9ac" path="/var/lib/kubelet/pods/ffd1bb54-c677-409c-93bf-6b5c244fa9ac/volumes"
Nov 02 01:33:17.340228 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:17.340179    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-4wp6c" podStartSLOduration=1.500506053 podStartE2EDuration="4.340161958s" totalImagesPullingTime="2.839655905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:33:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:33:17.339052808 +0000 UTC m=+462.242734921" watchObservedRunningTime="2026-11-02 01:33:17.340161958 +0000 UTC m=+462.243844073"
Nov 02 01:33:17.449256 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:17.449217    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nzbhb"]
Nov 02 01:33:17.451804 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:17.451778    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nzbhb"]
Nov 02 01:33:18.181742 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:18.181710    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-4wp6c"
Nov 02 01:33:19.217140 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:19.217108    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcaeb72e-96fa-429e-b5b6-f3ff966c93b2" path="/var/lib/kubelet/pods/bcaeb72e-96fa-429e-b5b6-f3ff966c93b2/volumes"
Nov 02 01:33:27.421847 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:33:27.421810    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:33:49.191824 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:33:49.191790    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-4wp6c"
Nov 02 01:33:57.419478 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:33:57.419437    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:34:27.419644 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:34:27.419607    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:34:57.419838 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:34:57.419754    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:35:27.419509 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:35:27.419471    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:35:30.799346 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.799314    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"]
Nov 02 01:35:30.799723 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.799665    5639 cpu_manager.go:414] "containerMap: removing container" podUID="bcaeb72e-96fa-429e-b5b6-f3ff966c93b2" containerName="authorino"
Nov 02 01:35:30.799723 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.799675    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcaeb72e-96fa-429e-b5b6-f3ff966c93b2" containerName="authorino"
Nov 02 01:35:30.799723 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.799685    5639 cpu_manager.go:414] "containerMap: removing container" podUID="e7a4933f-3f9c-4a30-80df-0a3d408fd0d6" containerName="authorino"
Nov 02 01:35:30.799723 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.799690    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7a4933f-3f9c-4a30-80df-0a3d408fd0d6" containerName="authorino"
Nov 02 01:35:30.799723 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.799712    5639 cpu_manager.go:414] "containerMap: removing container" podUID="ffd1bb54-c677-409c-93bf-6b5c244fa9ac" containerName="authorino"
Nov 02 01:35:30.799723 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.799717    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd1bb54-c677-409c-93bf-6b5c244fa9ac" containerName="authorino"
Nov 02 01:35:30.799912 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.799769    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcaeb72e-96fa-429e-b5b6-f3ff966c93b2" containerName="authorino"
Nov 02 01:35:30.799912 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.799776    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7a4933f-3f9c-4a30-80df-0a3d408fd0d6" containerName="authorino"
Nov 02 01:35:30.799912 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.799785    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd1bb54-c677-409c-93bf-6b5c244fa9ac" containerName="authorino"
Nov 02 01:35:30.802744 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.802728    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"
Nov 02 01:35:30.855691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.855662    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 02 01:35:30.877232 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.877200    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eccb539-67a3-4801-98bb-93a1ae520121-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f8b2k\" (UID: \"8eccb539-67a3-4801-98bb-93a1ae520121\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"
Nov 02 01:35:30.877232 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.877234    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eccb539-67a3-4801-98bb-93a1ae520121-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f8b2k\" (UID: \"8eccb539-67a3-4801-98bb-93a1ae520121\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"
Nov 02 01:35:30.877443 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.877266    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eccb539-67a3-4801-98bb-93a1ae520121-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f8b2k\" (UID: \"8eccb539-67a3-4801-98bb-93a1ae520121\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"
Nov 02 01:35:30.877485 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.877451    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eccb539-67a3-4801-98bb-93a1ae520121-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f8b2k\" (UID: \"8eccb539-67a3-4801-98bb-93a1ae520121\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"
Nov 02 01:35:30.877518 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.877484    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh2gx\" (UniqueName: \"kubernetes.io/projected/8eccb539-67a3-4801-98bb-93a1ae520121-kube-api-access-hh2gx\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f8b2k\" (UID: \"8eccb539-67a3-4801-98bb-93a1ae520121\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"
Nov 02 01:35:30.877558 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.877546    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eccb539-67a3-4801-98bb-93a1ae520121-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f8b2k\" (UID: \"8eccb539-67a3-4801-98bb-93a1ae520121\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"
Nov 02 01:35:30.904879 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.904857    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 02 01:35:30.905171 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.905152    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 02 01:35:30.905266 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.905232    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kf2fq\""
Nov 02 01:35:30.906544 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.906524    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"]
Nov 02 01:35:30.978967 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.978938    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eccb539-67a3-4801-98bb-93a1ae520121-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f8b2k\" (UID: \"8eccb539-67a3-4801-98bb-93a1ae520121\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"
Nov 02 01:35:30.979139 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.978997    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eccb539-67a3-4801-98bb-93a1ae520121-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f8b2k\" (UID: \"8eccb539-67a3-4801-98bb-93a1ae520121\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"
Nov 02 01:35:30.979139 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.979024    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eccb539-67a3-4801-98bb-93a1ae520121-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f8b2k\" (UID: \"8eccb539-67a3-4801-98bb-93a1ae520121\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"
Nov 02 01:35:30.979139 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.979050    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eccb539-67a3-4801-98bb-93a1ae520121-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f8b2k\" (UID: \"8eccb539-67a3-4801-98bb-93a1ae520121\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"
Nov 02 01:35:30.979139 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.979119    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eccb539-67a3-4801-98bb-93a1ae520121-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f8b2k\" (UID: \"8eccb539-67a3-4801-98bb-93a1ae520121\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"
Nov 02 01:35:30.979392 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.979150    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh2gx\" (UniqueName: \"kubernetes.io/projected/8eccb539-67a3-4801-98bb-93a1ae520121-kube-api-access-hh2gx\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f8b2k\" (UID: \"8eccb539-67a3-4801-98bb-93a1ae520121\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"
Nov 02 01:35:30.979453 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.979424    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eccb539-67a3-4801-98bb-93a1ae520121-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f8b2k\" (UID: \"8eccb539-67a3-4801-98bb-93a1ae520121\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"
Nov 02 01:35:30.979524 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.979493    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eccb539-67a3-4801-98bb-93a1ae520121-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f8b2k\" (UID: \"8eccb539-67a3-4801-98bb-93a1ae520121\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"
Nov 02 01:35:30.979581 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.979511    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eccb539-67a3-4801-98bb-93a1ae520121-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f8b2k\" (UID: \"8eccb539-67a3-4801-98bb-93a1ae520121\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"
Nov 02 01:35:30.981447 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.981427    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eccb539-67a3-4801-98bb-93a1ae520121-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f8b2k\" (UID: \"8eccb539-67a3-4801-98bb-93a1ae520121\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"
Nov 02 01:35:30.981698 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:30.981681    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eccb539-67a3-4801-98bb-93a1ae520121-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f8b2k\" (UID: \"8eccb539-67a3-4801-98bb-93a1ae520121\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"
Nov 02 01:35:31.040309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:31.040267    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh2gx\" (UniqueName: \"kubernetes.io/projected/8eccb539-67a3-4801-98bb-93a1ae520121-kube-api-access-hh2gx\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f8b2k\" (UID: \"8eccb539-67a3-4801-98bb-93a1ae520121\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"
Nov 02 01:35:31.115981 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:31.115949    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"
Nov 02 01:35:31.494703 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:31.494508    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"]
Nov 02 01:35:31.496865 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:35:31.496837    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eccb539_67a3_4801_98bb_93a1ae520121.slice/crio-b45a63373f8ac3d39a39449b314678ac01e71e212ff0b47700c91e501dd2c0ad WatchSource:0}: Error finding container b45a63373f8ac3d39a39449b314678ac01e71e212ff0b47700c91e501dd2c0ad: Status 404 returned error can't find the container with id b45a63373f8ac3d39a39449b314678ac01e71e212ff0b47700c91e501dd2c0ad
Nov 02 01:35:31.749102 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:31.749019    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k" event={"ID":"8eccb539-67a3-4801-98bb-93a1ae520121","Type":"ContainerStarted","Data":"b45a63373f8ac3d39a39449b314678ac01e71e212ff0b47700c91e501dd2c0ad"}
Nov 02 01:35:34.300224 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:34.300146    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:35:34.769089 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:34.769051    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k" event={"ID":"8eccb539-67a3-4801-98bb-93a1ae520121","Type":"ContainerStarted","Data":"4414574ec64115433056df2893d3026b057f25c3a3cb34a8a496dff77f13a5e3"}
Nov 02 01:35:34.895620 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:34.895567    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k" podStartSLOduration=2.55976996 podStartE2EDuration="4.895552559s" totalImagesPullingTime="2.335782599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:34.89432532 +0000 UTC m=+599.798007435" watchObservedRunningTime="2026-11-02 01:35:34.895552559 +0000 UTC m=+599.799234690"
Nov 02 01:35:35.116428 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:35.116391    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"
Nov 02 01:35:38.822717 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:38.822674    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"]
Nov 02 01:35:38.827048 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:38.827027    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"
Nov 02 01:35:38.928905 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:38.928877    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 02 01:35:38.930807 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:38.930777    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"]
Nov 02 01:35:38.949492 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:38.949459    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49af4db2-e792-4283-a393-47b5066464f8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8\" (UID: \"49af4db2-e792-4283-a393-47b5066464f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"
Nov 02 01:35:38.949641 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:38.949514    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7wnq\" (UniqueName: \"kubernetes.io/projected/49af4db2-e792-4283-a393-47b5066464f8-kube-api-access-b7wnq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8\" (UID: \"49af4db2-e792-4283-a393-47b5066464f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"
Nov 02 01:35:38.949682 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:38.949640    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49af4db2-e792-4283-a393-47b5066464f8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8\" (UID: \"49af4db2-e792-4283-a393-47b5066464f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"
Nov 02 01:35:38.949719 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:38.949703    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49af4db2-e792-4283-a393-47b5066464f8-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8\" (UID: \"49af4db2-e792-4283-a393-47b5066464f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"
Nov 02 01:35:38.949753 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:38.949738    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49af4db2-e792-4283-a393-47b5066464f8-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8\" (UID: \"49af4db2-e792-4283-a393-47b5066464f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"
Nov 02 01:35:38.949785 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:38.949753    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49af4db2-e792-4283-a393-47b5066464f8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8\" (UID: \"49af4db2-e792-4283-a393-47b5066464f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"
Nov 02 01:35:39.051096 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:39.051064    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49af4db2-e792-4283-a393-47b5066464f8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8\" (UID: \"49af4db2-e792-4283-a393-47b5066464f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"
Nov 02 01:35:39.051273 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:39.051104    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7wnq\" (UniqueName: \"kubernetes.io/projected/49af4db2-e792-4283-a393-47b5066464f8-kube-api-access-b7wnq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8\" (UID: \"49af4db2-e792-4283-a393-47b5066464f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"
Nov 02 01:35:39.051273 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:39.051136    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49af4db2-e792-4283-a393-47b5066464f8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8\" (UID: \"49af4db2-e792-4283-a393-47b5066464f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"
Nov 02 01:35:39.051273 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:39.051174    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49af4db2-e792-4283-a393-47b5066464f8-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8\" (UID: \"49af4db2-e792-4283-a393-47b5066464f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"
Nov 02 01:35:39.051273 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:39.051189    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49af4db2-e792-4283-a393-47b5066464f8-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8\" (UID: \"49af4db2-e792-4283-a393-47b5066464f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"
Nov 02 01:35:39.051273 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:39.051205    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49af4db2-e792-4283-a393-47b5066464f8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8\" (UID: \"49af4db2-e792-4283-a393-47b5066464f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"
Nov 02 01:35:39.051605 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:39.051580    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49af4db2-e792-4283-a393-47b5066464f8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8\" (UID: \"49af4db2-e792-4283-a393-47b5066464f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"
Nov 02 01:35:39.051605 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:39.051596    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49af4db2-e792-4283-a393-47b5066464f8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8\" (UID: \"49af4db2-e792-4283-a393-47b5066464f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"
Nov 02 01:35:39.051706 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:39.051648    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49af4db2-e792-4283-a393-47b5066464f8-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8\" (UID: \"49af4db2-e792-4283-a393-47b5066464f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"
Nov 02 01:35:39.053582 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:39.053563    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49af4db2-e792-4283-a393-47b5066464f8-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8\" (UID: \"49af4db2-e792-4283-a393-47b5066464f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"
Nov 02 01:35:39.053832 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:39.053816    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49af4db2-e792-4283-a393-47b5066464f8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8\" (UID: \"49af4db2-e792-4283-a393-47b5066464f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"
Nov 02 01:35:39.112012 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:39.111982    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7wnq\" (UniqueName: \"kubernetes.io/projected/49af4db2-e792-4283-a393-47b5066464f8-kube-api-access-b7wnq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8\" (UID: \"49af4db2-e792-4283-a393-47b5066464f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"
Nov 02 01:35:39.141109 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:39.141077    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"
Nov 02 01:35:39.512439 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:39.512359    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"]
Nov 02 01:35:39.515169 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:35:39.515145    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49af4db2_e792_4283_a393_47b5066464f8.slice/crio-a73a9cb1108f8c49091188123a3a930c9f4fc944bf1a15009b01d4d5b2bbf106 WatchSource:0}: Error finding container a73a9cb1108f8c49091188123a3a930c9f4fc944bf1a15009b01d4d5b2bbf106: Status 404 returned error can't find the container with id a73a9cb1108f8c49091188123a3a930c9f4fc944bf1a15009b01d4d5b2bbf106
Nov 02 01:35:39.792587 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:39.792507    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8" event={"ID":"49af4db2-e792-4283-a393-47b5066464f8","Type":"ContainerStarted","Data":"a73a9cb1108f8c49091188123a3a930c9f4fc944bf1a15009b01d4d5b2bbf106"}
Nov 02 01:35:40.798968 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:40.798938    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8" event={"ID":"49af4db2-e792-4283-a393-47b5066464f8","Type":"ContainerStarted","Data":"1c844a8e0227456d89ab60928bf96d4452bee6493ff99abaad5b5a70a083b88c"}
Nov 02 01:35:40.917020 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:40.916959    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8" podStartSLOduration=2.218097442 podStartE2EDuration="2.916930286s" totalImagesPullingTime="698.832844ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:40.916016638 +0000 UTC m=+605.819698753" watchObservedRunningTime="2026-11-02 01:35:40.916930286 +0000 UTC m=+605.820612407"
Nov 02 01:35:41.116872 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:41.116839    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"
Nov 02 01:35:41.129380 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:41.129357    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"
Nov 02 01:35:41.141640 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:41.141617    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"
Nov 02 01:35:41.814066 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:41.814035    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f8b2k"
Nov 02 01:35:42.604302 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:42.602894    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:35:42.879797 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:42.879726    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"]
Nov 02 01:35:42.883343 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:42.883326    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"
Nov 02 01:35:42.954851 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:42.954814    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 02 01:35:42.987047 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:42.987004    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"]
Nov 02 01:35:42.988386 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:42.988364    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/951363d0-8146-4109-8186-d232d1544131-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8jkg\" (UID: \"951363d0-8146-4109-8186-d232d1544131\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"
Nov 02 01:35:42.988442 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:42.988397    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/951363d0-8146-4109-8186-d232d1544131-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8jkg\" (UID: \"951363d0-8146-4109-8186-d232d1544131\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"
Nov 02 01:35:42.988509 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:42.988487    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/951363d0-8146-4109-8186-d232d1544131-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8jkg\" (UID: \"951363d0-8146-4109-8186-d232d1544131\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"
Nov 02 01:35:42.988546 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:42.988532    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/951363d0-8146-4109-8186-d232d1544131-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8jkg\" (UID: \"951363d0-8146-4109-8186-d232d1544131\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"
Nov 02 01:35:42.988668 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:42.988647    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/951363d0-8146-4109-8186-d232d1544131-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8jkg\" (UID: \"951363d0-8146-4109-8186-d232d1544131\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"
Nov 02 01:35:42.988769 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:42.988755    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkpm7\" (UniqueName: \"kubernetes.io/projected/951363d0-8146-4109-8186-d232d1544131-kube-api-access-nkpm7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8jkg\" (UID: \"951363d0-8146-4109-8186-d232d1544131\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"
Nov 02 01:35:43.089318 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:43.089260    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/951363d0-8146-4109-8186-d232d1544131-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8jkg\" (UID: \"951363d0-8146-4109-8186-d232d1544131\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"
Nov 02 01:35:43.089318 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:43.089315    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/951363d0-8146-4109-8186-d232d1544131-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8jkg\" (UID: \"951363d0-8146-4109-8186-d232d1544131\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"
Nov 02 01:35:43.089554 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:43.089360    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/951363d0-8146-4109-8186-d232d1544131-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8jkg\" (UID: \"951363d0-8146-4109-8186-d232d1544131\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"
Nov 02 01:35:43.089554 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:43.089483    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/951363d0-8146-4109-8186-d232d1544131-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8jkg\" (UID: \"951363d0-8146-4109-8186-d232d1544131\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"
Nov 02 01:35:43.089554 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:43.089551    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/951363d0-8146-4109-8186-d232d1544131-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8jkg\" (UID: \"951363d0-8146-4109-8186-d232d1544131\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"
Nov 02 01:35:43.089770 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:43.089641    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkpm7\" (UniqueName: \"kubernetes.io/projected/951363d0-8146-4109-8186-d232d1544131-kube-api-access-nkpm7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8jkg\" (UID: \"951363d0-8146-4109-8186-d232d1544131\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"
Nov 02 01:35:43.089823 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:43.089769    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/951363d0-8146-4109-8186-d232d1544131-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8jkg\" (UID: \"951363d0-8146-4109-8186-d232d1544131\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"
Nov 02 01:35:43.089823 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:43.089794    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/951363d0-8146-4109-8186-d232d1544131-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8jkg\" (UID: \"951363d0-8146-4109-8186-d232d1544131\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"
Nov 02 01:35:43.089928 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:43.089909    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/951363d0-8146-4109-8186-d232d1544131-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8jkg\" (UID: \"951363d0-8146-4109-8186-d232d1544131\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"
Nov 02 01:35:43.091542 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:43.091519    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/951363d0-8146-4109-8186-d232d1544131-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8jkg\" (UID: \"951363d0-8146-4109-8186-d232d1544131\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"
Nov 02 01:35:43.091834 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:43.091817    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/951363d0-8146-4109-8186-d232d1544131-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8jkg\" (UID: \"951363d0-8146-4109-8186-d232d1544131\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"
Nov 02 01:35:43.155916 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:43.155843    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkpm7\" (UniqueName: \"kubernetes.io/projected/951363d0-8146-4109-8186-d232d1544131-kube-api-access-nkpm7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8jkg\" (UID: \"951363d0-8146-4109-8186-d232d1544131\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"
Nov 02 01:35:43.199940 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:43.199899    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"
Nov 02 01:35:43.578863 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:43.578829    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"]
Nov 02 01:35:43.581952 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:35:43.581923    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod951363d0_8146_4109_8186_d232d1544131.slice/crio-af5610a623c38d1243350734417a2cef9450f8fcbd0ca13f130734a134b0ceba WatchSource:0}: Error finding container af5610a623c38d1243350734417a2cef9450f8fcbd0ca13f130734a134b0ceba: Status 404 returned error can't find the container with id af5610a623c38d1243350734417a2cef9450f8fcbd0ca13f130734a134b0ceba
Nov 02 01:35:43.812507 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:43.812465    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg" event={"ID":"951363d0-8146-4109-8186-d232d1544131","Type":"ContainerStarted","Data":"af5610a623c38d1243350734417a2cef9450f8fcbd0ca13f130734a134b0ceba"}
Nov 02 01:35:44.819450 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:44.819418    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg" event={"ID":"951363d0-8146-4109-8186-d232d1544131","Type":"ContainerStarted","Data":"a8ac18dc3ad6aea3494572f69a3e01dcff9787d4fb11388d9f0fbd68342144f6"}
Nov 02 01:35:44.940712 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:44.940660    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg" podStartSLOduration=2.228717693 podStartE2EDuration="2.940646453s" totalImagesPullingTime="711.92876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:44.939456913 +0000 UTC m=+609.843139028" watchObservedRunningTime="2026-11-02 01:35:44.940646453 +0000 UTC m=+609.844328566"
Nov 02 01:35:45.200849 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:45.200813    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"
Nov 02 01:35:49.141465 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:49.141427    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"
Nov 02 01:35:49.154380 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:49.154355    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"
Nov 02 01:35:49.852964 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:49.852936    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8"
Nov 02 01:35:52.210598 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:52.210562    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"]
Nov 02 01:35:52.214021 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:52.214005    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"
Nov 02 01:35:52.283990 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:52.283957    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b06646f-8efd-4ebc-9b9c-ff8f9734da72-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh\" (UID: \"6b06646f-8efd-4ebc-9b9c-ff8f9734da72\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"
Nov 02 01:35:52.284157 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:52.283995    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twntx\" (UniqueName: \"kubernetes.io/projected/6b06646f-8efd-4ebc-9b9c-ff8f9734da72-kube-api-access-twntx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh\" (UID: \"6b06646f-8efd-4ebc-9b9c-ff8f9734da72\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"
Nov 02 01:35:52.284157 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:52.284025    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b06646f-8efd-4ebc-9b9c-ff8f9734da72-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh\" (UID: \"6b06646f-8efd-4ebc-9b9c-ff8f9734da72\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"
Nov 02 01:35:52.284157 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:52.284055    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b06646f-8efd-4ebc-9b9c-ff8f9734da72-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh\" (UID: \"6b06646f-8efd-4ebc-9b9c-ff8f9734da72\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"
Nov 02 01:35:52.284263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:52.284232    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b06646f-8efd-4ebc-9b9c-ff8f9734da72-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh\" (UID: \"6b06646f-8efd-4ebc-9b9c-ff8f9734da72\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"
Nov 02 01:35:52.284323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:52.284273    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b06646f-8efd-4ebc-9b9c-ff8f9734da72-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh\" (UID: \"6b06646f-8efd-4ebc-9b9c-ff8f9734da72\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"
Nov 02 01:35:52.315894 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:52.315864    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 02 01:35:52.317210 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:52.317191    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"]
Nov 02 01:35:52.385002 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:52.384969    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b06646f-8efd-4ebc-9b9c-ff8f9734da72-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh\" (UID: \"6b06646f-8efd-4ebc-9b9c-ff8f9734da72\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"
Nov 02 01:35:52.385002 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:52.385007    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twntx\" (UniqueName: \"kubernetes.io/projected/6b06646f-8efd-4ebc-9b9c-ff8f9734da72-kube-api-access-twntx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh\" (UID: \"6b06646f-8efd-4ebc-9b9c-ff8f9734da72\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"
Nov 02 01:35:52.385228 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:52.385036    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b06646f-8efd-4ebc-9b9c-ff8f9734da72-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh\" (UID: \"6b06646f-8efd-4ebc-9b9c-ff8f9734da72\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"
Nov 02 01:35:52.385228 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:52.385068    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b06646f-8efd-4ebc-9b9c-ff8f9734da72-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh\" (UID: \"6b06646f-8efd-4ebc-9b9c-ff8f9734da72\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"
Nov 02 01:35:52.385228 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:52.385108    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b06646f-8efd-4ebc-9b9c-ff8f9734da72-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh\" (UID: \"6b06646f-8efd-4ebc-9b9c-ff8f9734da72\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"
Nov 02 01:35:52.385228 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:52.385151    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b06646f-8efd-4ebc-9b9c-ff8f9734da72-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh\" (UID: \"6b06646f-8efd-4ebc-9b9c-ff8f9734da72\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"
Nov 02 01:35:52.385474 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:52.385427    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b06646f-8efd-4ebc-9b9c-ff8f9734da72-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh\" (UID: \"6b06646f-8efd-4ebc-9b9c-ff8f9734da72\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"
Nov 02 01:35:52.385535 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:52.385511    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b06646f-8efd-4ebc-9b9c-ff8f9734da72-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh\" (UID: \"6b06646f-8efd-4ebc-9b9c-ff8f9734da72\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"
Nov 02 01:35:52.385583 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:52.385550    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b06646f-8efd-4ebc-9b9c-ff8f9734da72-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh\" (UID: \"6b06646f-8efd-4ebc-9b9c-ff8f9734da72\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"
Nov 02 01:35:52.387604 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:52.387583    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b06646f-8efd-4ebc-9b9c-ff8f9734da72-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh\" (UID: \"6b06646f-8efd-4ebc-9b9c-ff8f9734da72\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"
Nov 02 01:35:52.387770 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:52.387752    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b06646f-8efd-4ebc-9b9c-ff8f9734da72-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh\" (UID: \"6b06646f-8efd-4ebc-9b9c-ff8f9734da72\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"
Nov 02 01:35:52.446622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:52.446588    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twntx\" (UniqueName: \"kubernetes.io/projected/6b06646f-8efd-4ebc-9b9c-ff8f9734da72-kube-api-access-twntx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh\" (UID: \"6b06646f-8efd-4ebc-9b9c-ff8f9734da72\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"
Nov 02 01:35:52.526565 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:52.526485    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"
Nov 02 01:35:52.923045 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:52.923015    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"]
Nov 02 01:35:52.928366 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:35:52.928339    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b06646f_8efd_4ebc_9b9c_ff8f9734da72.slice/crio-ea1cdf05d1eef85c714a49ae7afce8a8355856efb7b5e010bd468973dbc31fe5 WatchSource:0}: Error finding container ea1cdf05d1eef85c714a49ae7afce8a8355856efb7b5e010bd468973dbc31fe5: Status 404 returned error can't find the container with id ea1cdf05d1eef85c714a49ae7afce8a8355856efb7b5e010bd468973dbc31fe5
Nov 02 01:35:52.994304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:52.993460    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:35:53.200959 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.200869    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"
Nov 02 01:35:53.217365 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.217339    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"
Nov 02 01:35:53.460012 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.459926    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"]
Nov 02 01:35:53.464758 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.464741    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.552215 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.552178    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"]
Nov 02 01:35:53.600537 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.600504    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pjf7g\""
Nov 02 01:35:53.600705 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.600578    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.600705 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.600606    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.600705 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.600634    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.600813 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.600735    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.600813 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.600767    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.600813 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.600790    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182a2c2c-a34a-41c8-bf80-982d15b15d26-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.600905 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.600862    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.600941 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.600925    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8zdn\" (UniqueName: \"kubernetes.io/projected/182a2c2c-a34a-41c8-bf80-982d15b15d26-kube-api-access-f8zdn\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.600987 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.600973    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.702121 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.702095    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8zdn\" (UniqueName: \"kubernetes.io/projected/182a2c2c-a34a-41c8-bf80-982d15b15d26-kube-api-access-f8zdn\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.702243 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.702182    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.702243 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.702233    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.702375 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.702256    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.702696 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.702670    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.703063 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.703034    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.703165 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.703101    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.703165 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.703142    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.703961 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.703181    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182a2c2c-a34a-41c8-bf80-982d15b15d26-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.703961 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.703247    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.703961 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.703534    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.703961 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.703616    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.703961 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.703797    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.704687 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.704662    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182a2c2c-a34a-41c8-bf80-982d15b15d26-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.704877 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.704853    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.705363 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.705342    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.763081 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.763015    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.763217 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.763130    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8zdn\" (UniqueName: \"kubernetes.io/projected/182a2c2c-a34a-41c8-bf80-982d15b15d26-kube-api-access-f8zdn\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t7rwt\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.779031 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.778992    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:53.862639 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.862609    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh" event={"ID":"6b06646f-8efd-4ebc-9b9c-ff8f9734da72","Type":"ContainerStarted","Data":"2587f19ea5b361082fb63dc5f5eabbee00f2037a74a16996e9d17fc22cb89aba"}
Nov 02 01:35:53.862796 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.862663    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh" event={"ID":"6b06646f-8efd-4ebc-9b9c-ff8f9734da72","Type":"ContainerStarted","Data":"ea1cdf05d1eef85c714a49ae7afce8a8355856efb7b5e010bd468973dbc31fe5"}
Nov 02 01:35:53.873758 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.873730    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8jkg"
Nov 02 01:35:53.981306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:53.980536    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh" podStartSLOduration=1.264136711 podStartE2EDuration="1.980517854s" totalImagesPullingTime="716.381143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:53.980165971 +0000 UTC m=+618.883848078" watchObservedRunningTime="2026-11-02 01:35:53.980517854 +0000 UTC m=+618.884199968"
Nov 02 01:35:54.204067 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:54.204043    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"]
Nov 02 01:35:54.206722 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:35:54.206690    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182a2c2c_a34a_41c8_bf80_982d15b15d26.slice/crio-c1c02f78bd7004160ba33e16d4f8fa249c3e4143c78aace1bd2a6f8ed80f8ad4 WatchSource:0}: Error finding container c1c02f78bd7004160ba33e16d4f8fa249c3e4143c78aace1bd2a6f8ed80f8ad4: Status 404 returned error can't find the container with id c1c02f78bd7004160ba33e16d4f8fa249c3e4143c78aace1bd2a6f8ed80f8ad4
Nov 02 01:35:54.526709 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:54.526616    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"
Nov 02 01:35:54.869187 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:54.869150    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt" event={"ID":"182a2c2c-a34a-41c8-bf80-982d15b15d26","Type":"ContainerStarted","Data":"c1c02f78bd7004160ba33e16d4f8fa249c3e4143c78aace1bd2a6f8ed80f8ad4"}
Nov 02 01:35:57.422848 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:35:57.422803    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:35:58.327968 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:58.327935    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:35:58.328058 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:58.328009    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:35:58.328058 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:58.328039    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:35:58.890956 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:58.890921    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt" event={"ID":"182a2c2c-a34a-41c8-bf80-982d15b15d26","Type":"ContainerStarted","Data":"660e7907ff8148d7166b8f378cee54932601f86b79ae4c3195bf8bcad4c2e3bf"}
Nov 02 01:35:59.073305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:59.065680    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt" podStartSLOduration=1.9475221820000002 podStartE2EDuration="6.065661323s" totalImagesPullingTime="4.118139141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:59.0645765 +0000 UTC m=+623.968258615" watchObservedRunningTime="2026-11-02 01:35:59.065661323 +0000 UTC m=+623.969343439"
Nov 02 01:35:59.784365 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:59.779606    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:35:59.785920 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:59.785882    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 01:35:59.786029 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:35:59.785945    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt" podUID="182a2c2c-a34a-41c8-bf80-982d15b15d26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 01:36:00.306762 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:00.306708    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"]
Nov 02 01:36:00.312430 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:00.312404    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"
Nov 02 01:36:00.387940 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:00.387426    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"]
Nov 02 01:36:00.415600 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:00.415568    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 02 01:36:00.489631 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:00.489580    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/351876aa-c801-49f4-acdc-667422401ce3-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq\" (UID: \"351876aa-c801-49f4-acdc-667422401ce3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"
Nov 02 01:36:00.489631 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:00.489623    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/351876aa-c801-49f4-acdc-667422401ce3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq\" (UID: \"351876aa-c801-49f4-acdc-667422401ce3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"
Nov 02 01:36:00.489631 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:00.489644    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/351876aa-c801-49f4-acdc-667422401ce3-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq\" (UID: \"351876aa-c801-49f4-acdc-667422401ce3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"
Nov 02 01:36:00.489954 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:00.489691    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/351876aa-c801-49f4-acdc-667422401ce3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq\" (UID: \"351876aa-c801-49f4-acdc-667422401ce3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"
Nov 02 01:36:00.489954 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:00.489713    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7glpl\" (UniqueName: \"kubernetes.io/projected/351876aa-c801-49f4-acdc-667422401ce3-kube-api-access-7glpl\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq\" (UID: \"351876aa-c801-49f4-acdc-667422401ce3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"
Nov 02 01:36:00.489954 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:00.489734    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/351876aa-c801-49f4-acdc-667422401ce3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq\" (UID: \"351876aa-c801-49f4-acdc-667422401ce3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"
Nov 02 01:36:00.590903 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:00.590865    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/351876aa-c801-49f4-acdc-667422401ce3-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq\" (UID: \"351876aa-c801-49f4-acdc-667422401ce3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"
Nov 02 01:36:00.591062 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:00.590935    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/351876aa-c801-49f4-acdc-667422401ce3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq\" (UID: \"351876aa-c801-49f4-acdc-667422401ce3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"
Nov 02 01:36:00.591062 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:00.590954    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7glpl\" (UniqueName: \"kubernetes.io/projected/351876aa-c801-49f4-acdc-667422401ce3-kube-api-access-7glpl\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq\" (UID: \"351876aa-c801-49f4-acdc-667422401ce3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"
Nov 02 01:36:00.591062 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:00.590971    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/351876aa-c801-49f4-acdc-667422401ce3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq\" (UID: \"351876aa-c801-49f4-acdc-667422401ce3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"
Nov 02 01:36:00.591161 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:00.591062    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/351876aa-c801-49f4-acdc-667422401ce3-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq\" (UID: \"351876aa-c801-49f4-acdc-667422401ce3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"
Nov 02 01:36:00.591161 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:00.591088    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/351876aa-c801-49f4-acdc-667422401ce3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq\" (UID: \"351876aa-c801-49f4-acdc-667422401ce3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"
Nov 02 01:36:00.591500 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:00.591468    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/351876aa-c801-49f4-acdc-667422401ce3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq\" (UID: \"351876aa-c801-49f4-acdc-667422401ce3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"
Nov 02 01:36:00.591648 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:00.591530    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/351876aa-c801-49f4-acdc-667422401ce3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq\" (UID: \"351876aa-c801-49f4-acdc-667422401ce3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"
Nov 02 01:36:00.591648 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:00.591620    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/351876aa-c801-49f4-acdc-667422401ce3-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq\" (UID: \"351876aa-c801-49f4-acdc-667422401ce3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"
Nov 02 01:36:00.593433 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:00.593412    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/351876aa-c801-49f4-acdc-667422401ce3-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq\" (UID: \"351876aa-c801-49f4-acdc-667422401ce3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"
Nov 02 01:36:00.593684 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:00.593665    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/351876aa-c801-49f4-acdc-667422401ce3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq\" (UID: \"351876aa-c801-49f4-acdc-667422401ce3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"
Nov 02 01:36:00.652373 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:00.652332    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7glpl\" (UniqueName: \"kubernetes.io/projected/351876aa-c801-49f4-acdc-667422401ce3-kube-api-access-7glpl\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq\" (UID: \"351876aa-c801-49f4-acdc-667422401ce3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"
Nov 02 01:36:00.780756 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:00.780709    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 01:36:00.781043 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:00.781011    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt" podUID="182a2c2c-a34a-41c8-bf80-982d15b15d26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 01:36:00.926366 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:00.925467    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"
Nov 02 01:36:01.422263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:01.422230    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"]
Nov 02 01:36:01.424774 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:36:01.424738    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351876aa_c801_49f4_acdc_667422401ce3.slice/crio-d363a5db61ab4a861ea08b12a389729b150c55a2a7c99cdaf0c4ca50ec8825f0 WatchSource:0}: Error finding container d363a5db61ab4a861ea08b12a389729b150c55a2a7c99cdaf0c4ca50ec8825f0: Status 404 returned error can't find the container with id d363a5db61ab4a861ea08b12a389729b150c55a2a7c99cdaf0c4ca50ec8825f0
Nov 02 01:36:01.782211 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:01.781701    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 01:36:01.782211 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:01.781782    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt" podUID="182a2c2c-a34a-41c8-bf80-982d15b15d26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 01:36:01.907837 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:01.907798    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq" event={"ID":"351876aa-c801-49f4-acdc-667422401ce3","Type":"ContainerStarted","Data":"d363a5db61ab4a861ea08b12a389729b150c55a2a7c99cdaf0c4ca50ec8825f0"}
Nov 02 01:36:02.527901 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:02.527807    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"
Nov 02 01:36:02.541066 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:02.541036    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"
Nov 02 01:36:02.784867 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:02.784772    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 01:36:02.785318 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:02.785126    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt" podUID="182a2c2c-a34a-41c8-bf80-982d15b15d26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 01:36:02.913987 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:02.913953    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq" event={"ID":"351876aa-c801-49f4-acdc-667422401ce3","Type":"ContainerStarted","Data":"9b7645c15046e4d19e2864d7bb8b82094e303d3669d8101cf53245eb095c2459"}
Nov 02 01:36:02.925376 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:02.925350    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh"
Nov 02 01:36:02.926461 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:02.926443    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"
Nov 02 01:36:03.033242 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:03.033175    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq" podStartSLOduration=2.323749088 podStartE2EDuration="3.033155944s" totalImagesPullingTime="709.406856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:03.032244101 +0000 UTC m=+627.935926215" watchObservedRunningTime="2026-11-02 01:36:03.033155944 +0000 UTC m=+627.936838060"
Nov 02 01:36:03.784329 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:03.780745    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 01:36:03.784329 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:03.780807    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt" podUID="182a2c2c-a34a-41c8-bf80-982d15b15d26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 01:36:04.779963 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:04.779926    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 01:36:04.780138 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:04.779984    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt" podUID="182a2c2c-a34a-41c8-bf80-982d15b15d26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 01:36:06.821454 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:06.821409    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:36:06.821968 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:06.821489    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt" podUID="182a2c2c-a34a-41c8-bf80-982d15b15d26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:36:07.602048 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:07.601860    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:36:07.602548 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:07.602521    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:36:07.603243 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:07.603224    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:36:08.491889 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:08.491856    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:36:10.925954 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:10.925922    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"
Nov 02 01:36:10.938225 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:10.938200    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"
Nov 02 01:36:10.964904 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:10.964879    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq"
Nov 02 01:36:27.419716 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:36:27.419633    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:36:27.729874 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:27.729783    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-595dbbb7d4-4gbj4"]
Nov 02 01:36:27.733478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:27.733455    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-595dbbb7d4-4gbj4"
Nov 02 01:36:27.790427 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:27.790397    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-595dbbb7d4-4gbj4"]
Nov 02 01:36:27.847837 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:27.847794    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ffb99aaa-ca56-4f11-a26a-20d80226df6a-tls-cert\") pod \"authorino-595dbbb7d4-4gbj4\" (UID: \"ffb99aaa-ca56-4f11-a26a-20d80226df6a\") " pod="kuadrant-system/authorino-595dbbb7d4-4gbj4"
Nov 02 01:36:27.848007 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:27.847925    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdzkr\" (UniqueName: \"kubernetes.io/projected/ffb99aaa-ca56-4f11-a26a-20d80226df6a-kube-api-access-cdzkr\") pod \"authorino-595dbbb7d4-4gbj4\" (UID: \"ffb99aaa-ca56-4f11-a26a-20d80226df6a\") " pod="kuadrant-system/authorino-595dbbb7d4-4gbj4"
Nov 02 01:36:27.949165 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:27.949130    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ffb99aaa-ca56-4f11-a26a-20d80226df6a-tls-cert\") pod \"authorino-595dbbb7d4-4gbj4\" (UID: \"ffb99aaa-ca56-4f11-a26a-20d80226df6a\") " pod="kuadrant-system/authorino-595dbbb7d4-4gbj4"
Nov 02 01:36:27.949371 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:27.949189    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdzkr\" (UniqueName: \"kubernetes.io/projected/ffb99aaa-ca56-4f11-a26a-20d80226df6a-kube-api-access-cdzkr\") pod \"authorino-595dbbb7d4-4gbj4\" (UID: \"ffb99aaa-ca56-4f11-a26a-20d80226df6a\") " pod="kuadrant-system/authorino-595dbbb7d4-4gbj4"
Nov 02 01:36:27.952125 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:27.952103    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ffb99aaa-ca56-4f11-a26a-20d80226df6a-tls-cert\") pod \"authorino-595dbbb7d4-4gbj4\" (UID: \"ffb99aaa-ca56-4f11-a26a-20d80226df6a\") " pod="kuadrant-system/authorino-595dbbb7d4-4gbj4"
Nov 02 01:36:28.010316 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:28.010222    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdzkr\" (UniqueName: \"kubernetes.io/projected/ffb99aaa-ca56-4f11-a26a-20d80226df6a-kube-api-access-cdzkr\") pod \"authorino-595dbbb7d4-4gbj4\" (UID: \"ffb99aaa-ca56-4f11-a26a-20d80226df6a\") " pod="kuadrant-system/authorino-595dbbb7d4-4gbj4"
Nov 02 01:36:28.045604 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:28.045543    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-595dbbb7d4-4gbj4"
Nov 02 01:36:28.421626 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:28.421588    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-595dbbb7d4-4gbj4"]
Nov 02 01:36:28.424595 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:36:28.424567    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffb99aaa_ca56_4f11_a26a_20d80226df6a.slice/crio-ab000757299d4911c72bea14609291ced2fcb7ecdb665b1b190f9ff51cac88e5 WatchSource:0}: Error finding container ab000757299d4911c72bea14609291ced2fcb7ecdb665b1b190f9ff51cac88e5: Status 404 returned error can't find the container with id ab000757299d4911c72bea14609291ced2fcb7ecdb665b1b190f9ff51cac88e5
Nov 02 01:36:29.035568 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:29.035534    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-595dbbb7d4-4gbj4" event={"ID":"ffb99aaa-ca56-4f11-a26a-20d80226df6a","Type":"ContainerStarted","Data":"ab000757299d4911c72bea14609291ced2fcb7ecdb665b1b190f9ff51cac88e5"}
Nov 02 01:36:31.047096 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:31.047052    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-595dbbb7d4-4gbj4" event={"ID":"ffb99aaa-ca56-4f11-a26a-20d80226df6a","Type":"ContainerStarted","Data":"3df688e5e71a0bba45f54fabce01a5bc88d23e530b1afa1752780f3df34cb6f7"}
Nov 02 01:36:31.162723 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:31.162658    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-595dbbb7d4-4gbj4" podStartSLOduration=2.543891325 podStartE2EDuration="4.162638493s" totalImagesPullingTime="1.618747168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:31.16176816 +0000 UTC m=+656.065450275" watchObservedRunningTime="2026-11-02 01:36:31.162638493 +0000 UTC m=+656.066320608"
Nov 02 01:36:31.221081 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:31.221047    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687b947b6c-w5459"]
Nov 02 01:36:31.221353 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:31.221325    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-687b947b6c-w5459" podUID="cb62a2e9-fc33-49ed-a2b9-9d46c8305edf" containerName="authorino" containerID="cri-o://301c38c9108984c25e9c031e9f27baa0f5edc2269479b4c180c02e658a826b23" gracePeriod=30
Nov 02 01:36:31.476749 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:31.476720    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687b947b6c-w5459"
Nov 02 01:36:31.587362 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:31.586580    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb62a2e9-fc33-49ed-a2b9-9d46c8305edf-tls-cert\" (UniqueName: \"kubernetes.io/secret/cb62a2e9-fc33-49ed-a2b9-9d46c8305edf-tls-cert\") pod \"cb62a2e9-fc33-49ed-a2b9-9d46c8305edf\" (UID: \"cb62a2e9-fc33-49ed-a2b9-9d46c8305edf\") "
Nov 02 01:36:31.587362 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:31.586757    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb62a2e9-fc33-49ed-a2b9-9d46c8305edf-kube-api-access-k496c\" (UniqueName: \"kubernetes.io/projected/cb62a2e9-fc33-49ed-a2b9-9d46c8305edf-kube-api-access-k496c\") pod \"cb62a2e9-fc33-49ed-a2b9-9d46c8305edf\" (UID: \"cb62a2e9-fc33-49ed-a2b9-9d46c8305edf\") "
Nov 02 01:36:31.597645 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:31.597609    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb62a2e9-fc33-49ed-a2b9-9d46c8305edf-kube-api-access-k496c" pod "cb62a2e9-fc33-49ed-a2b9-9d46c8305edf" (UID: "cb62a2e9-fc33-49ed-a2b9-9d46c8305edf"). InnerVolumeSpecName "kube-api-access-k496c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:36:31.608404 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:31.608369    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb62a2e9-fc33-49ed-a2b9-9d46c8305edf-tls-cert" pod "cb62a2e9-fc33-49ed-a2b9-9d46c8305edf" (UID: "cb62a2e9-fc33-49ed-a2b9-9d46c8305edf"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:36:31.688230 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:31.688193    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k496c\" (UniqueName: \"kubernetes.io/projected/cb62a2e9-fc33-49ed-a2b9-9d46c8305edf-kube-api-access-k496c\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:36:31.688230 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:31.688227    5639 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb62a2e9-fc33-49ed-a2b9-9d46c8305edf-tls-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:36:32.067081 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:32.066983    5639 generic.go:356] "Generic (PLEG): container finished" podID="cb62a2e9-fc33-49ed-a2b9-9d46c8305edf" containerID="301c38c9108984c25e9c031e9f27baa0f5edc2269479b4c180c02e658a826b23" exitCode=0
Nov 02 01:36:32.067081 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:32.067033    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687b947b6c-w5459"
Nov 02 01:36:32.067081 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:32.067070    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-687b947b6c-w5459" event={"ID":"cb62a2e9-fc33-49ed-a2b9-9d46c8305edf","Type":"ContainerDied","Data":"301c38c9108984c25e9c031e9f27baa0f5edc2269479b4c180c02e658a826b23"}
Nov 02 01:36:32.067673 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:32.067119    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-687b947b6c-w5459" event={"ID":"cb62a2e9-fc33-49ed-a2b9-9d46c8305edf","Type":"ContainerDied","Data":"86d63186b5aa29ac1d6b4be6a103baf745148b823b82e3e7db0b96b54054e7ea"}
Nov 02 01:36:32.067673 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:32.067139    5639 scope.go:122] "RemoveContainer" containerID="301c38c9108984c25e9c031e9f27baa0f5edc2269479b4c180c02e658a826b23"
Nov 02 01:36:32.078646 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:32.078626    5639 scope.go:122] "RemoveContainer" containerID="301c38c9108984c25e9c031e9f27baa0f5edc2269479b4c180c02e658a826b23"
Nov 02 01:36:32.078893 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:36:32.078873    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"301c38c9108984c25e9c031e9f27baa0f5edc2269479b4c180c02e658a826b23\": container with ID starting with 301c38c9108984c25e9c031e9f27baa0f5edc2269479b4c180c02e658a826b23 not found: ID does not exist" containerID="301c38c9108984c25e9c031e9f27baa0f5edc2269479b4c180c02e658a826b23"
Nov 02 01:36:32.078939 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:32.078910    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"301c38c9108984c25e9c031e9f27baa0f5edc2269479b4c180c02e658a826b23"} err="failed to get container status \"301c38c9108984c25e9c031e9f27baa0f5edc2269479b4c180c02e658a826b23\": rpc error: code = NotFound desc = could not find container \"301c38c9108984c25e9c031e9f27baa0f5edc2269479b4c180c02e658a826b23\": container with ID starting with 301c38c9108984c25e9c031e9f27baa0f5edc2269479b4c180c02e658a826b23 not found: ID does not exist"
Nov 02 01:36:32.230070 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:32.230025    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687b947b6c-w5459"]
Nov 02 01:36:32.279330 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:32.279300    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687b947b6c-w5459"]
Nov 02 01:36:33.218063 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:33.218028    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb62a2e9-fc33-49ed-a2b9-9d46c8305edf" path="/var/lib/kubelet/pods/cb62a2e9-fc33-49ed-a2b9-9d46c8305edf/volumes"
Nov 02 01:36:37.662613 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:37.662580    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-smrvz"]
Nov 02 01:36:37.663002 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:37.662990    5639 cpu_manager.go:414] "containerMap: removing container" podUID="cb62a2e9-fc33-49ed-a2b9-9d46c8305edf" containerName="authorino"
Nov 02 01:36:37.663042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:37.663004    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb62a2e9-fc33-49ed-a2b9-9d46c8305edf" containerName="authorino"
Nov 02 01:36:37.663082 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:37.663063    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb62a2e9-fc33-49ed-a2b9-9d46c8305edf" containerName="authorino"
Nov 02 01:36:37.665698 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:37.665678    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-smrvz"
Nov 02 01:36:37.717912 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:37.717882    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 02 01:36:37.718058 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:37.717921    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-rk85l\""
Nov 02 01:36:37.724132 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:37.724106    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-smrvz"]
Nov 02 01:36:37.745703 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:37.745669    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c574ce35-1740-4193-bcc3-38a38425b040-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-smrvz\" (UID: \"c574ce35-1740-4193-bcc3-38a38425b040\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-smrvz"
Nov 02 01:36:37.745855 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:37.745753    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q8bp\" (UniqueName: \"kubernetes.io/projected/c574ce35-1740-4193-bcc3-38a38425b040-kube-api-access-9q8bp\") pod \"payload-pre-processing-7b98bb678c-smrvz\" (UID: \"c574ce35-1740-4193-bcc3-38a38425b040\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-smrvz"
Nov 02 01:36:37.821358 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:37.821322    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-c276v"]
Nov 02 01:36:37.824392 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:37.824366    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-c276v"
Nov 02 01:36:37.847088 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:37.847063    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q8bp\" (UniqueName: \"kubernetes.io/projected/c574ce35-1740-4193-bcc3-38a38425b040-kube-api-access-9q8bp\") pod \"payload-pre-processing-7b98bb678c-smrvz\" (UID: \"c574ce35-1740-4193-bcc3-38a38425b040\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-smrvz"
Nov 02 01:36:37.847260 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:37.847149    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c574ce35-1740-4193-bcc3-38a38425b040-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-smrvz\" (UID: \"c574ce35-1740-4193-bcc3-38a38425b040\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-smrvz"
Nov 02 01:36:37.847887 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:37.847863    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c574ce35-1740-4193-bcc3-38a38425b040-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-smrvz\" (UID: \"c574ce35-1740-4193-bcc3-38a38425b040\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-smrvz"
Nov 02 01:36:37.881042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:37.881020    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-c276v"]
Nov 02 01:36:37.908019 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:37.907995    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q8bp\" (UniqueName: \"kubernetes.io/projected/c574ce35-1740-4193-bcc3-38a38425b040-kube-api-access-9q8bp\") pod \"payload-pre-processing-7b98bb678c-smrvz\" (UID: \"c574ce35-1740-4193-bcc3-38a38425b040\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-smrvz"
Nov 02 01:36:37.948468 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:37.948397    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a75b836c-cfb5-4f2b-9a49-ae3b350b08d0-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-c276v\" (UID: \"a75b836c-cfb5-4f2b-9a49-ae3b350b08d0\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-c276v"
Nov 02 01:36:37.948468 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:37.948441    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5pc5\" (UniqueName: \"kubernetes.io/projected/a75b836c-cfb5-4f2b-9a49-ae3b350b08d0-kube-api-access-h5pc5\") pod \"payload-processing-55fbbbf9dc-c276v\" (UID: \"a75b836c-cfb5-4f2b-9a49-ae3b350b08d0\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-c276v"
Nov 02 01:36:37.977326 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:37.977299    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-smrvz"
Nov 02 01:36:38.049909 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:38.049876    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a75b836c-cfb5-4f2b-9a49-ae3b350b08d0-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-c276v\" (UID: \"a75b836c-cfb5-4f2b-9a49-ae3b350b08d0\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-c276v"
Nov 02 01:36:38.050048 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:38.049946    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5pc5\" (UniqueName: \"kubernetes.io/projected/a75b836c-cfb5-4f2b-9a49-ae3b350b08d0-kube-api-access-h5pc5\") pod \"payload-processing-55fbbbf9dc-c276v\" (UID: \"a75b836c-cfb5-4f2b-9a49-ae3b350b08d0\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-c276v"
Nov 02 01:36:38.050842 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:38.050819    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a75b836c-cfb5-4f2b-9a49-ae3b350b08d0-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-c276v\" (UID: \"a75b836c-cfb5-4f2b-9a49-ae3b350b08d0\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-c276v"
Nov 02 01:36:38.111016 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:38.110988    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5pc5\" (UniqueName: \"kubernetes.io/projected/a75b836c-cfb5-4f2b-9a49-ae3b350b08d0-kube-api-access-h5pc5\") pod \"payload-processing-55fbbbf9dc-c276v\" (UID: \"a75b836c-cfb5-4f2b-9a49-ae3b350b08d0\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-c276v"
Nov 02 01:36:38.137875 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:38.137842    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-c276v"
Nov 02 01:36:38.347524 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:38.347499    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-smrvz"]
Nov 02 01:36:38.349492 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:36:38.349466    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc574ce35_1740_4193_bcc3_38a38425b040.slice/crio-f09822232c6d72c12e2536c124301e29aeea8e2a89e8e6ce071442c6c68d0d1f WatchSource:0}: Error finding container f09822232c6d72c12e2536c124301e29aeea8e2a89e8e6ce071442c6c68d0d1f: Status 404 returned error can't find the container with id f09822232c6d72c12e2536c124301e29aeea8e2a89e8e6ce071442c6c68d0d1f
Nov 02 01:36:38.506836 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:38.506806    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-c276v"]
Nov 02 01:36:38.509374 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:36:38.509338    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75b836c_cfb5_4f2b_9a49_ae3b350b08d0.slice/crio-acc4f90af2e91a6dc6c88691964e0042efc3cae9c2395050f613621b07342b70 WatchSource:0}: Error finding container acc4f90af2e91a6dc6c88691964e0042efc3cae9c2395050f613621b07342b70: Status 404 returned error can't find the container with id acc4f90af2e91a6dc6c88691964e0042efc3cae9c2395050f613621b07342b70
Nov 02 01:36:39.101539 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:39.101501    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-c276v" event={"ID":"a75b836c-cfb5-4f2b-9a49-ae3b350b08d0","Type":"ContainerStarted","Data":"acc4f90af2e91a6dc6c88691964e0042efc3cae9c2395050f613621b07342b70"}
Nov 02 01:36:39.102719 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:39.102696    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-smrvz" event={"ID":"c574ce35-1740-4193-bcc3-38a38425b040","Type":"ContainerStarted","Data":"f09822232c6d72c12e2536c124301e29aeea8e2a89e8e6ce071442c6c68d0d1f"}
Nov 02 01:36:42.119044 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:42.119014    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-c276v" event={"ID":"a75b836c-cfb5-4f2b-9a49-ae3b350b08d0","Type":"ContainerStarted","Data":"ecfb90482c5dee9572b6cf39bbb7e7da882d37612da4be51d2e3d1b346359234"}
Nov 02 01:36:42.119469 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:42.119138    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-c276v"
Nov 02 01:36:42.120711 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:42.120689    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-smrvz" event={"ID":"c574ce35-1740-4193-bcc3-38a38425b040","Type":"ContainerStarted","Data":"ccb70620920261ad94ee037d7ce5c2f3b778b1f8e4d92f6cb1d0e7e0a2ad4f04"}
Nov 02 01:36:42.120834 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:42.120748    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-smrvz"
Nov 02 01:36:42.235109 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:42.235054    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-c276v" podStartSLOduration=1.712565653 podStartE2EDuration="5.235041266s" totalImagesPullingTime="3.522475613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:42.234086757 +0000 UTC m=+667.137768871" watchObservedRunningTime="2026-11-02 01:36:42.235041266 +0000 UTC m=+667.138723393"
Nov 02 01:36:42.350226 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:42.350161    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-smrvz" podStartSLOduration=1.986296749 podStartE2EDuration="5.35014051s" totalImagesPullingTime="3.363843761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:42.348688466 +0000 UTC m=+667.252370580" watchObservedRunningTime="2026-11-02 01:36:42.35014051 +0000 UTC m=+667.253822625"
Nov 02 01:36:53.128488 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:53.128460    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-smrvz"
Nov 02 01:36:53.128939 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:53.128523    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-c276v"
Nov 02 01:36:53.936143 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:53.936113    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"]
Nov 02 01:36:53.939535 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:53.939519    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.043646 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.043614    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"]
Nov 02 01:36:54.104163 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.104126    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db2e4537-de21-48f8-858e-1ead87636161-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.104163 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.104165    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2e4537-de21-48f8-858e-1ead87636161-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.104419 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.104239    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db2e4537-de21-48f8-858e-1ead87636161-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.104419 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.104350    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2e4537-de21-48f8-858e-1ead87636161-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.104419 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.104397    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db2e4537-de21-48f8-858e-1ead87636161-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.104419 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.104419    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db2e4537-de21-48f8-858e-1ead87636161-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.104572 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.104451    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db2e4537-de21-48f8-858e-1ead87636161-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.104572 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.104480    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db2e4537-de21-48f8-858e-1ead87636161-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.104572 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.104548    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn2nc\" (UniqueName: \"kubernetes.io/projected/db2e4537-de21-48f8-858e-1ead87636161-kube-api-access-tn2nc\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.205803 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.205727    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2e4537-de21-48f8-858e-1ead87636161-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.205803 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.205772    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db2e4537-de21-48f8-858e-1ead87636161-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.206424 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.205807    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db2e4537-de21-48f8-858e-1ead87636161-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.206424 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.205853    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db2e4537-de21-48f8-858e-1ead87636161-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.206424 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.205878    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db2e4537-de21-48f8-858e-1ead87636161-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.206424 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.205907    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn2nc\" (UniqueName: \"kubernetes.io/projected/db2e4537-de21-48f8-858e-1ead87636161-kube-api-access-tn2nc\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.206424 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.205970    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db2e4537-de21-48f8-858e-1ead87636161-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.206424 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.206000    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2e4537-de21-48f8-858e-1ead87636161-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.206424 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.206044    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db2e4537-de21-48f8-858e-1ead87636161-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.206424 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.206198    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db2e4537-de21-48f8-858e-1ead87636161-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.206424 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.206392    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2e4537-de21-48f8-858e-1ead87636161-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.206813 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.206452    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db2e4537-de21-48f8-858e-1ead87636161-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.206813 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.206583    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2e4537-de21-48f8-858e-1ead87636161-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.206813 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.206695    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db2e4537-de21-48f8-858e-1ead87636161-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.208444 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.208422    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db2e4537-de21-48f8-858e-1ead87636161-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.209046 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.209029    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db2e4537-de21-48f8-858e-1ead87636161-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.267191 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.267160    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db2e4537-de21-48f8-858e-1ead87636161-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.267350 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.267222    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn2nc\" (UniqueName: \"kubernetes.io/projected/db2e4537-de21-48f8-858e-1ead87636161-kube-api-access-tn2nc\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q6wpx\" (UID: \"db2e4537-de21-48f8-858e-1ead87636161\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.552184 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.552109    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:54.978259 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.978235    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"]
Nov 02 01:36:54.980433 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:36:54.980406    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb2e4537_de21_48f8_858e_1ead87636161.slice/crio-48d498dffaae5902091cfe1cd25592c698417497b5dd07aa85652f4bc7f58afc WatchSource:0}: Error finding container 48d498dffaae5902091cfe1cd25592c698417497b5dd07aa85652f4bc7f58afc: Status 404 returned error can't find the container with id 48d498dffaae5902091cfe1cd25592c698417497b5dd07aa85652f4bc7f58afc
Nov 02 01:36:54.982687 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.982650    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:36:54.982793 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.982728    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:36:54.982793 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:54.982771    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:36:55.182857 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:55.182818    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx" event={"ID":"db2e4537-de21-48f8-858e-1ead87636161","Type":"ContainerStarted","Data":"3ec065fea1705517e377229eabb13eddac059afd6e8ea9ce6d2a8984a4bae993"}
Nov 02 01:36:55.183046 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:55.182877    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx" event={"ID":"db2e4537-de21-48f8-858e-1ead87636161","Type":"ContainerStarted","Data":"48d498dffaae5902091cfe1cd25592c698417497b5dd07aa85652f4bc7f58afc"}
Nov 02 01:36:55.304684 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:55.304595    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx" podStartSLOduration=2.3045812740000002 podStartE2EDuration="2.304581274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:55.30299031 +0000 UTC m=+680.206672424" watchObservedRunningTime="2026-11-02 01:36:55.304581274 +0000 UTC m=+680.208263448"
Nov 02 01:36:55.552732 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:55.552701    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:36:56.554569 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:56.554536    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 02 01:36:56.554954 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:56.554591    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx" podUID="db2e4537-de21-48f8-858e-1ead87636161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 02 01:36:57.419775 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:36:57.419743    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:36:57.553363 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:57.553328    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 02 01:36:57.553533 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:57.553388    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx" podUID="db2e4537-de21-48f8-858e-1ead87636161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 02 01:36:58.553209 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:58.553177    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 02 01:36:58.553595 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:58.553236    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx" podUID="db2e4537-de21-48f8-858e-1ead87636161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 02 01:36:59.552881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:59.552849    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 02 01:36:59.553054 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:36:59.552901    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx" podUID="db2e4537-de21-48f8-858e-1ead87636161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 02 01:37:00.552639 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:00.552607    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 02 01:37:00.553022 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:00.552669    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx" podUID="db2e4537-de21-48f8-858e-1ead87636161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 02 01:37:01.553237 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:01.553200    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 02 01:37:01.553680 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:01.553256    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx" podUID="db2e4537-de21-48f8-858e-1ead87636161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 02 01:37:03.210421 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:03.210386    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:37:03.216877 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:03.216855    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:37:03.217030 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:03.216914    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx"
Nov 02 01:37:27.420170 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:37:27.420131    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:37:34.688900 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.688866    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"]
Nov 02 01:37:34.692773 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.692754    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.747863 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.747812    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b71ba5-openshift-default-dockercfg-bzhxb\""
Nov 02 01:37:34.798862 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.798831    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"]
Nov 02 01:37:34.819015 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.818985    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.819173 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.819033    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb685866-3bb8-43ae-85f9-17cea849a9f8-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.819173 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.819048    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m7tk\" (UniqueName: \"kubernetes.io/projected/eb685866-3bb8-43ae-85f9-17cea849a9f8-kube-api-access-2m7tk\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.819173 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.819141    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.819291 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.819237    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.819291 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.819262    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.819370 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.819357    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.819407 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.819380    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.819442 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.819405    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.920087 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.920061    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.920087 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.920092    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.920346 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.920119    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.920346 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.920140    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.920346 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.920193    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb685866-3bb8-43ae-85f9-17cea849a9f8-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.920346 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.920216    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m7tk\" (UniqueName: \"kubernetes.io/projected/eb685866-3bb8-43ae-85f9-17cea849a9f8-kube-api-access-2m7tk\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.920346 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.920255    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.920346 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.920335    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.920668 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.920366    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.920668 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.920564    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.920668 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.920618    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.920818 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.920732    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.920818 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.920726    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.920921 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.920884    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb685866-3bb8-43ae-85f9-17cea849a9f8-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.922802 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.922777    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.923122 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.923104    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.981004 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.980927    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:34.981138 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:34.981102    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m7tk\" (UniqueName: \"kubernetes.io/projected/eb685866-3bb8-43ae-85f9-17cea849a9f8-kube-api-access-2m7tk\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:35.004204 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.004176    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"]
Nov 02 01:37:35.006725 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.006702    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:35.009133 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.009115    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.063338 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.063314    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-187478-openshift-default-dockercfg-gsqtc\""
Nov 02 01:37:35.070888 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.070236    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"]
Nov 02 01:37:35.121732 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.121694    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.121883 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.121737    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.121883 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.121762    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.121883 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.121878    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t67jl\" (UniqueName: \"kubernetes.io/projected/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-kube-api-access-t67jl\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.122003 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.121926    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.122003 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.121970    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.122003 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.121993    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.122121 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.122047    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.122177 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.122155    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.222904 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.222874    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.223060 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.222915    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.223060 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.222989    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.223060 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.223033    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.223200 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.223100    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.223200 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.223130    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.223311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.223217    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.223311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.223248    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t67jl\" (UniqueName: \"kubernetes.io/projected/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-kube-api-access-t67jl\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.223311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.223269    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.223534 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.223510    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.223808 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.223787    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.223882 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.223843    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.223947 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.223900    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.224086 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.224063    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.225475 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.225459    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.225475 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.225467    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.284146 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.284078    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t67jl\" (UniqueName: \"kubernetes.io/projected/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-kube-api-access-t67jl\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.284346 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.284270    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-626p2\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.384104 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.384072    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-187478-openshift-default-dockercfg-gsqtc\""
Nov 02 01:37:35.384293 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.384167    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:35.446920 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.446895    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"]
Nov 02 01:37:35.452069 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:37:35.452043    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb685866_3bb8_43ae_85f9_17cea849a9f8.slice/crio-769efe5c877019b84790972fb1e060ec4e968a1c37963afd6271295d71206624 WatchSource:0}: Error finding container 769efe5c877019b84790972fb1e060ec4e968a1c37963afd6271295d71206624: Status 404 returned error can't find the container with id 769efe5c877019b84790972fb1e060ec4e968a1c37963afd6271295d71206624
Nov 02 01:37:35.454496 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.454464    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:37:35.454601 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.454526    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:37:35.454601 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.454556    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:37:35.763361 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.763334    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"]
Nov 02 01:37:35.765925 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:37:35.765892    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4dec6db_d72f_45ce_802c_f54bb7f57e5c.slice/crio-2764475bdd37450dc8b3639d6f5c79d579b6a2835826cb63dd615473eabe96f7 WatchSource:0}: Error finding container 2764475bdd37450dc8b3639d6f5c79d579b6a2835826cb63dd615473eabe96f7: Status 404 returned error can't find the container with id 2764475bdd37450dc8b3639d6f5c79d579b6a2835826cb63dd615473eabe96f7
Nov 02 01:37:35.768770 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.768738    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:37:35.768864 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.768801    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:37:35.768864 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.768829    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:37:35.974947 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.974917    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"]
Nov 02 01:37:35.978548 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:35.978527    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.035153 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.035054    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-88e76d-openshift-default-dockercfg-gsdsg\""
Nov 02 01:37:36.042128 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.041609    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"]
Nov 02 01:37:36.143409 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.142351    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.143409 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.142396    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.143409 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.142443    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.143409 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.142572    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.143409 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.142640    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kscz\" (UniqueName: \"kubernetes.io/projected/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-kube-api-access-8kscz\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.143409 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.142779    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.143409 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.142805    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.143409 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.142845    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.143409 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.142914    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.244114 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.244081    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.244114 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.244119    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.244386 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.244149    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.244386 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.244195    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.244386 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.244235    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.244386 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.244252    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.244386 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.244307    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.244386 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.244335    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.244386 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.244376    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kscz\" (UniqueName: \"kubernetes.io/projected/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-kube-api-access-8kscz\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.244724 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.244616    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.244724 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.244684    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.244819 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.244727    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.244819 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.244750    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.245166 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.245148    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.246895 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.246855    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.247208 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.247188    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.305634 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.305563    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kscz\" (UniqueName: \"kubernetes.io/projected/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-kube-api-access-8kscz\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.305875 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.305854    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.322856 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.322817    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"]
Nov 02 01:37:36.327547 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.327528    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.381463 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.381423    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp" event={"ID":"eb685866-3bb8-43ae-85f9-17cea849a9f8","Type":"ContainerStarted","Data":"8d940a2ad7728f0ce218a8285737b15ad93fdda1762f17cfa23c346b8ade176c"}
Nov 02 01:37:36.381658 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.381526    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp" event={"ID":"eb685866-3bb8-43ae-85f9-17cea849a9f8","Type":"ContainerStarted","Data":"769efe5c877019b84790972fb1e060ec4e968a1c37963afd6271295d71206624"}
Nov 02 01:37:36.383945 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.383920    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2" event={"ID":"c4dec6db-d72f-45ce-802c-f54bb7f57e5c","Type":"ContainerStarted","Data":"686c26bc415f61deefbc110374f8064e90e1eb57966696875b12338d48775fbd"}
Nov 02 01:37:36.384178 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.383958    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2" event={"ID":"c4dec6db-d72f-45ce-802c-f54bb7f57e5c","Type":"ContainerStarted","Data":"2764475bdd37450dc8b3639d6f5c79d579b6a2835826cb63dd615473eabe96f7"}
Nov 02 01:37:36.390609 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.390554    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:36.390815 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.390795    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"]
Nov 02 01:37:36.396765 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.396742    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:36.446199 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.446163    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf946ac-2c79-4b61-8890-703068477cdf-istio-podinfo\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.446504 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.446216    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-credential-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.446504 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.446401    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-workload-certs\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.446991 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.446781    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-istio-data\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.446991 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.446898    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmcr6\" (UniqueName: \"kubernetes.io/projected/8bf946ac-2c79-4b61-8890-703068477cdf-kube-api-access-pmcr6\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.447077 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.447035    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-istio-envoy\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.447152 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.447135    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf946ac-2c79-4b61-8890-703068477cdf-istiod-ca-cert\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.447204 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.447163    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-workload-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.447259 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.447201    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf946ac-2c79-4b61-8890-703068477cdf-istio-token\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.458569 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.458548    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-36a1e0-openshift-default-dockercfg-lck7x\""
Nov 02 01:37:36.547954 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.547916    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-istio-envoy\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.548150 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.548057    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf946ac-2c79-4b61-8890-703068477cdf-istiod-ca-cert\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.548150 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.548087    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-workload-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.548150 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.548139    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf946ac-2c79-4b61-8890-703068477cdf-istio-token\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.548358 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.548187    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf946ac-2c79-4b61-8890-703068477cdf-istio-podinfo\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.548358 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.548218    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-credential-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.548358 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.548290    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-workload-certs\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.548518 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.548367    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-istio-data\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.548518 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.548393    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmcr6\" (UniqueName: \"kubernetes.io/projected/8bf946ac-2c79-4b61-8890-703068477cdf-kube-api-access-pmcr6\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.548624 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.548552    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-credential-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.548975 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.548947    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-workload-certs\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.549107 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.549087    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-istio-data\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.549203 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.549177    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf946ac-2c79-4b61-8890-703068477cdf-istiod-ca-cert\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.549323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.549181    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-workload-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.550581 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.550564    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-istio-envoy\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.550980 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.550955    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf946ac-2c79-4b61-8890-703068477cdf-istio-podinfo\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.567625 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.567175    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp" podStartSLOduration=2.567161125 podStartE2EDuration="2.567161125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:36.566896285 +0000 UTC m=+721.470578400" watchObservedRunningTime="2026-11-02 01:37:36.567161125 +0000 UTC m=+721.470843241"
Nov 02 01:37:36.591629 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.591602    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:36.609529 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.609502    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf946ac-2c79-4b61-8890-703068477cdf-istio-token\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.609635 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.609589    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmcr6\" (UniqueName: \"kubernetes.io/projected/8bf946ac-2c79-4b61-8890-703068477cdf-kube-api-access-pmcr6\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.643786 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.641946    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:36.969604 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.969553    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2" podStartSLOduration=2.9695370309999998 podStartE2EDuration="2.969537031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:36.700826297 +0000 UTC m=+721.604508432" watchObservedRunningTime="2026-11-02 01:37:36.969537031 +0000 UTC m=+721.873219140"
Nov 02 01:37:36.971203 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.971181    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"]
Nov 02 01:37:36.972927 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:37:36.972902    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da5e48a_b3fd_4b3a_97d9_505aea24dc94.slice/crio-eb179c3ee8425c2ddfd9c45b85c59c57fd349c9660905e15af50afa17074e735 WatchSource:0}: Error finding container eb179c3ee8425c2ddfd9c45b85c59c57fd349c9660905e15af50afa17074e735: Status 404 returned error can't find the container with id eb179c3ee8425c2ddfd9c45b85c59c57fd349c9660905e15af50afa17074e735
Nov 02 01:37:36.975365 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.975333    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:37:36.975469 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.975411    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:37:36.975538 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:36.975492    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:37:37.007649 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:37.007620    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:37.013002 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:37.012976    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:37.048272 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:37.048250    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"]
Nov 02 01:37:37.051023 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:37:37.050993    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf946ac_2c79_4b61_8890_703068477cdf.slice/crio-b3bdbd4c37cb5f97a7d0256257a4cc640b28d1f27d8c80fed69317ea08011e0f WatchSource:0}: Error finding container b3bdbd4c37cb5f97a7d0256257a4cc640b28d1f27d8c80fed69317ea08011e0f: Status 404 returned error can't find the container with id b3bdbd4c37cb5f97a7d0256257a4cc640b28d1f27d8c80fed69317ea08011e0f
Nov 02 01:37:37.053465 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:37.053430    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:37:37.053545 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:37.053512    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:37:37.053589 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:37.053556    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:37:37.391830 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:37.391791    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm" event={"ID":"8bf946ac-2c79-4b61-8890-703068477cdf","Type":"ContainerStarted","Data":"e95cf8fbdc7e541966df1c6e1eeb15cbc21cc51e2caacb56439a668f1d97211d"}
Nov 02 01:37:37.392003 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:37.391861    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm" event={"ID":"8bf946ac-2c79-4b61-8890-703068477cdf","Type":"ContainerStarted","Data":"b3bdbd4c37cb5f97a7d0256257a4cc640b28d1f27d8c80fed69317ea08011e0f"}
Nov 02 01:37:37.393789 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:37.393756    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq" event={"ID":"2da5e48a-b3fd-4b3a-97d9-505aea24dc94","Type":"ContainerStarted","Data":"fd7ec3605ebe009a0d9fb2a852381c7e698fc9de83f5d8bfd0f56ccc28447ef7"}
Nov 02 01:37:37.394190 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:37.394075    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:37.394190 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:37.394129    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:37.394190 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:37.394153    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq" event={"ID":"2da5e48a-b3fd-4b3a-97d9-505aea24dc94","Type":"ContainerStarted","Data":"eb179c3ee8425c2ddfd9c45b85c59c57fd349c9660905e15af50afa17074e735"}
Nov 02 01:37:37.394830 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:37.394808    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:37:37.394924 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:37.394864    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:37:37.517926 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:37.517877    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm" podStartSLOduration=1.5178636650000001 podStartE2EDuration="1.517863665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:37.516388042 +0000 UTC m=+722.420070165" watchObservedRunningTime="2026-11-02 01:37:37.517863665 +0000 UTC m=+722.421545777"
Nov 02 01:37:37.592031 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:37.592003    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:37.642031 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:37.641918    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq" podStartSLOduration=2.641896442 podStartE2EDuration="2.641896442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:37.640682759 +0000 UTC m=+722.544364872" watchObservedRunningTime="2026-11-02 01:37:37.641896442 +0000 UTC m=+722.545578567"
Nov 02 01:37:37.648618 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:37.643678    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:38.597430 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:38.597402    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:38.647692 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:38.647661    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:39.405574 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:39.405542    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:39.405766 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:39.405599    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:39.406251 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:39.406232    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:37:39.406373 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:39.406306    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:37:57.364785 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:57.364713    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g"]
Nov 02 01:37:57.368198 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:57.368179    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g"
Nov 02 01:37:57.419846 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:37:57.419815    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:37:57.469792 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:57.469765    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-e06ea2-dockercfg-sbglj\""
Nov 02 01:37:57.470061 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:57.470043    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-e06ea2\""
Nov 02 01:37:57.471747 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:57.471729    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g"]
Nov 02 01:37:57.483784 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:57.483757    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e96d87f-8aab-450c-8865-b7daec4f91a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g\" (UID: \"2e96d87f-8aab-450c-8865-b7daec4f91a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g"
Nov 02 01:37:57.483920 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:57.483792    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79vjc\" (UniqueName: \"kubernetes.io/projected/2e96d87f-8aab-450c-8865-b7daec4f91a5-kube-api-access-79vjc\") pod \"payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g\" (UID: \"2e96d87f-8aab-450c-8865-b7daec4f91a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g"
Nov 02 01:37:57.585381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:57.585349    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e96d87f-8aab-450c-8865-b7daec4f91a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g\" (UID: \"2e96d87f-8aab-450c-8865-b7daec4f91a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g"
Nov 02 01:37:57.585381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:57.585379    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79vjc\" (UniqueName: \"kubernetes.io/projected/2e96d87f-8aab-450c-8865-b7daec4f91a5-kube-api-access-79vjc\") pod \"payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g\" (UID: \"2e96d87f-8aab-450c-8865-b7daec4f91a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g"
Nov 02 01:37:57.586005 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:57.585985    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e96d87f-8aab-450c-8865-b7daec4f91a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g\" (UID: \"2e96d87f-8aab-450c-8865-b7daec4f91a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g"
Nov 02 01:37:57.645981 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:57.645901    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79vjc\" (UniqueName: \"kubernetes.io/projected/2e96d87f-8aab-450c-8865-b7daec4f91a5-kube-api-access-79vjc\") pod \"payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g\" (UID: \"2e96d87f-8aab-450c-8865-b7daec4f91a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g"
Nov 02 01:37:57.681002 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:57.680974    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g"
Nov 02 01:37:58.052404 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:58.052200    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g"]
Nov 02 01:37:58.054620 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:37:58.054590    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e96d87f_8aab_450c_8865_b7daec4f91a5.slice/crio-cd92539acb0d8449ca7e1e3803d7d5a7a3960469ac0bf29160df3a5564b67781 WatchSource:0}: Error finding container cd92539acb0d8449ca7e1e3803d7d5a7a3960469ac0bf29160df3a5564b67781: Status 404 returned error can't find the container with id cd92539acb0d8449ca7e1e3803d7d5a7a3960469ac0bf29160df3a5564b67781
Nov 02 01:37:58.498975 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:58.498940    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g" event={"ID":"2e96d87f-8aab-450c-8865-b7daec4f91a5","Type":"ContainerStarted","Data":"ca08b6b9822fde2c76f0670856e6b948de5ac92a55b4400aaee83f678c8bf9bb"}
Nov 02 01:37:58.499490 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:58.499032    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g"
Nov 02 01:37:58.499490 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:58.499053    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g" event={"ID":"2e96d87f-8aab-450c-8865-b7daec4f91a5","Type":"ContainerStarted","Data":"cd92539acb0d8449ca7e1e3803d7d5a7a3960469ac0bf29160df3a5564b67781"}
Nov 02 01:37:58.881320 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:58.878809    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g" podStartSLOduration=1.878787118 podStartE2EDuration="1.878787118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:58.617870513 +0000 UTC m=+743.521552638" watchObservedRunningTime="2026-11-02 01:37:58.878787118 +0000 UTC m=+743.782469233"
Nov 02 01:37:58.881320 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:37:58.879958    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:38:02.262309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:02.257370    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd"]
Nov 02 01:38:02.262801 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:02.262362    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd"
Nov 02 01:38:02.298980 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:02.298939    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd"]
Nov 02 01:38:02.346255 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:02.346225    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-36a1e0\""
Nov 02 01:38:02.346506 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:02.346487    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-36a1e0-dockercfg-kjsnv\""
Nov 02 01:38:02.362968 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:02.362931    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5qr2\" (UniqueName: \"kubernetes.io/projected/7e5a265a-31f7-4042-88cc-a18783a14ecb-kube-api-access-v5qr2\") pod \"payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd\" (UID: \"7e5a265a-31f7-4042-88cc-a18783a14ecb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd"
Nov 02 01:38:02.363159 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:02.363138    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5a265a-31f7-4042-88cc-a18783a14ecb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd\" (UID: \"7e5a265a-31f7-4042-88cc-a18783a14ecb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd"
Nov 02 01:38:02.464717 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:02.464685    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5a265a-31f7-4042-88cc-a18783a14ecb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd\" (UID: \"7e5a265a-31f7-4042-88cc-a18783a14ecb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd"
Nov 02 01:38:02.464922 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:02.464763    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5qr2\" (UniqueName: \"kubernetes.io/projected/7e5a265a-31f7-4042-88cc-a18783a14ecb-kube-api-access-v5qr2\") pod \"payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd\" (UID: \"7e5a265a-31f7-4042-88cc-a18783a14ecb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd"
Nov 02 01:38:02.465408 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:02.465387    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5a265a-31f7-4042-88cc-a18783a14ecb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd\" (UID: \"7e5a265a-31f7-4042-88cc-a18783a14ecb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd"
Nov 02 01:38:02.525913 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:02.525846    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5qr2\" (UniqueName: \"kubernetes.io/projected/7e5a265a-31f7-4042-88cc-a18783a14ecb-kube-api-access-v5qr2\") pod \"payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd\" (UID: \"7e5a265a-31f7-4042-88cc-a18783a14ecb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd"
Nov 02 01:38:02.579324 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:02.579267    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd"
Nov 02 01:38:02.950349 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:02.950266    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd"]
Nov 02 01:38:02.953065 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:38:02.953034    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5a265a_31f7_4042_88cc_a18783a14ecb.slice/crio-cb86a80b369ee768b100040f424e35577f5bbe6fcff65808c7d670110d4f3fda WatchSource:0}: Error finding container cb86a80b369ee768b100040f424e35577f5bbe6fcff65808c7d670110d4f3fda: Status 404 returned error can't find the container with id cb86a80b369ee768b100040f424e35577f5bbe6fcff65808c7d670110d4f3fda
Nov 02 01:38:03.525667 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:03.525621    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd" event={"ID":"7e5a265a-31f7-4042-88cc-a18783a14ecb","Type":"ContainerStarted","Data":"d406f0b5df890b98c315f9e3a59237bc9bf3213c0ec931117c0dbd959c6162ec"}
Nov 02 01:38:03.526105 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:03.525687    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd" event={"ID":"7e5a265a-31f7-4042-88cc-a18783a14ecb","Type":"ContainerStarted","Data":"cb86a80b369ee768b100040f424e35577f5bbe6fcff65808c7d670110d4f3fda"}
Nov 02 01:38:03.526105 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:03.525709    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd"
Nov 02 01:38:03.641947 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:03.641893    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd" podStartSLOduration=1.641878736 podStartE2EDuration="1.641878736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:03.640410518 +0000 UTC m=+748.544092660" watchObservedRunningTime="2026-11-02 01:38:03.641878736 +0000 UTC m=+748.545560849"
Nov 02 01:38:07.280355 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:07.280322    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2"]
Nov 02 01:38:07.284197 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:07.284180    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2"
Nov 02 01:38:07.339901 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:07.339641    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-f276c3\""
Nov 02 01:38:07.388986 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:07.388942    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-f276c3-dockercfg-kjpcl\""
Nov 02 01:38:07.390990 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:07.390966    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2"]
Nov 02 01:38:07.419915 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:07.419889    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2ed806-9940-4872-93b2-0cd86f5aeb52-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2\" (UID: \"2d2ed806-9940-4872-93b2-0cd86f5aeb52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2"
Nov 02 01:38:07.420057 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:07.419928    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwvbv\" (UniqueName: \"kubernetes.io/projected/2d2ed806-9940-4872-93b2-0cd86f5aeb52-kube-api-access-dwvbv\") pod \"payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2\" (UID: \"2d2ed806-9940-4872-93b2-0cd86f5aeb52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2"
Nov 02 01:38:07.521441 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:07.521407    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2ed806-9940-4872-93b2-0cd86f5aeb52-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2\" (UID: \"2d2ed806-9940-4872-93b2-0cd86f5aeb52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2"
Nov 02 01:38:07.521621 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:07.521449    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwvbv\" (UniqueName: \"kubernetes.io/projected/2d2ed806-9940-4872-93b2-0cd86f5aeb52-kube-api-access-dwvbv\") pod \"payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2\" (UID: \"2d2ed806-9940-4872-93b2-0cd86f5aeb52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2"
Nov 02 01:38:07.522161 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:07.522141    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2ed806-9940-4872-93b2-0cd86f5aeb52-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2\" (UID: \"2d2ed806-9940-4872-93b2-0cd86f5aeb52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2"
Nov 02 01:38:07.582472 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:07.582442    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwvbv\" (UniqueName: \"kubernetes.io/projected/2d2ed806-9940-4872-93b2-0cd86f5aeb52-kube-api-access-dwvbv\") pod \"payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2\" (UID: \"2d2ed806-9940-4872-93b2-0cd86f5aeb52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2"
Nov 02 01:38:07.597156 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:07.597129    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2"
Nov 02 01:38:07.969020 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:07.968998    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2"]
Nov 02 01:38:07.971270 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:38:07.971237    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2ed806_9940_4872_93b2_0cd86f5aeb52.slice/crio-b035945b06e93855d72451e7a360dc41f611c470391c170d1ba0c66fbab27da3 WatchSource:0}: Error finding container b035945b06e93855d72451e7a360dc41f611c470391c170d1ba0c66fbab27da3: Status 404 returned error can't find the container with id b035945b06e93855d72451e7a360dc41f611c470391c170d1ba0c66fbab27da3
Nov 02 01:38:08.267071 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:08.266991    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:38:08.552237 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:08.552152    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2" event={"ID":"2d2ed806-9940-4872-93b2-0cd86f5aeb52","Type":"ContainerStarted","Data":"91d58044a9e6dc09bfb24c95c48c3b1a073ca0ac0b669fe87a092960fcca328f"}
Nov 02 01:38:08.552237 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:08.552200    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2" event={"ID":"2d2ed806-9940-4872-93b2-0cd86f5aeb52","Type":"ContainerStarted","Data":"b035945b06e93855d72451e7a360dc41f611c470391c170d1ba0c66fbab27da3"}
Nov 02 01:38:08.552687 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:08.552297    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2"
Nov 02 01:38:09.508009 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:09.507978    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g"
Nov 02 01:38:09.627940 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:09.627875    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2" podStartSLOduration=2.627855246 podStartE2EDuration="2.627855246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:08.680251361 +0000 UTC m=+753.583933489" watchObservedRunningTime="2026-11-02 01:38:09.627855246 +0000 UTC m=+754.531537362"
Nov 02 01:38:11.939134 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:11.939103    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6"]
Nov 02 01:38:11.943855 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:11.943839    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6"
Nov 02 01:38:11.998326 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:11.998256    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-88e76d\""
Nov 02 01:38:12.050417 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:12.050381    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-88e76d-dockercfg-xgbfr\""
Nov 02 01:38:12.052736 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:12.052697    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6"]
Nov 02 01:38:12.103317 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:12.103237    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24"]
Nov 02 01:38:12.107418 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:12.107396    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24"
Nov 02 01:38:12.107925 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:12.107900    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b145a144-84c0-43c6-b08b-c5c196efddf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6\" (UID: \"b145a144-84c0-43c6-b08b-c5c196efddf9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6"
Nov 02 01:38:12.108100 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:12.108076    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7lkx\" (UniqueName: \"kubernetes.io/projected/b145a144-84c0-43c6-b08b-c5c196efddf9-kube-api-access-c7lkx\") pod \"payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6\" (UID: \"b145a144-84c0-43c6-b08b-c5c196efddf9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6"
Nov 02 01:38:12.146570 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:12.146536    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24"]
Nov 02 01:38:12.209729 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:12.209647    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b145a144-84c0-43c6-b08b-c5c196efddf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6\" (UID: \"b145a144-84c0-43c6-b08b-c5c196efddf9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6"
Nov 02 01:38:12.209729 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:12.209687    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7lkx\" (UniqueName: \"kubernetes.io/projected/b145a144-84c0-43c6-b08b-c5c196efddf9-kube-api-access-c7lkx\") pod \"payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6\" (UID: \"b145a144-84c0-43c6-b08b-c5c196efddf9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6"
Nov 02 01:38:12.209927 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:12.209750    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6p66\" (UniqueName: \"kubernetes.io/projected/5b302fb0-288a-4d1d-aece-d87bdf645f92-kube-api-access-q6p66\") pod \"payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24\" (UID: \"5b302fb0-288a-4d1d-aece-d87bdf645f92\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24"
Nov 02 01:38:12.209927 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:12.209772    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b302fb0-288a-4d1d-aece-d87bdf645f92-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24\" (UID: \"5b302fb0-288a-4d1d-aece-d87bdf645f92\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24"
Nov 02 01:38:12.210409 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:12.210390    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b145a144-84c0-43c6-b08b-c5c196efddf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6\" (UID: \"b145a144-84c0-43c6-b08b-c5c196efddf9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6"
Nov 02 01:38:12.286788 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:12.286753    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7lkx\" (UniqueName: \"kubernetes.io/projected/b145a144-84c0-43c6-b08b-c5c196efddf9-kube-api-access-c7lkx\") pod \"payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6\" (UID: \"b145a144-84c0-43c6-b08b-c5c196efddf9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6"
Nov 02 01:38:12.310819 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:12.310787    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6p66\" (UniqueName: \"kubernetes.io/projected/5b302fb0-288a-4d1d-aece-d87bdf645f92-kube-api-access-q6p66\") pod \"payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24\" (UID: \"5b302fb0-288a-4d1d-aece-d87bdf645f92\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24"
Nov 02 01:38:12.310819 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:12.310826    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b302fb0-288a-4d1d-aece-d87bdf645f92-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24\" (UID: \"5b302fb0-288a-4d1d-aece-d87bdf645f92\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24"
Nov 02 01:38:12.311521 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:12.311500    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b302fb0-288a-4d1d-aece-d87bdf645f92-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24\" (UID: \"5b302fb0-288a-4d1d-aece-d87bdf645f92\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24"
Nov 02 01:38:12.385167 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:12.384623    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6p66\" (UniqueName: \"kubernetes.io/projected/5b302fb0-288a-4d1d-aece-d87bdf645f92-kube-api-access-q6p66\") pod \"payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24\" (UID: \"5b302fb0-288a-4d1d-aece-d87bdf645f92\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24"
Nov 02 01:38:12.422872 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:12.422838    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24"
Nov 02 01:38:12.556922 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:12.556890    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6"
Nov 02 01:38:12.830783 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:12.830759    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24"]
Nov 02 01:38:12.833007 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:38:12.832977    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b302fb0_288a_4d1d_aece_d87bdf645f92.slice/crio-91b9e9985a17c8f97681ff21aaa60a26af8ea0a6a591b130398ed2a9c2f0e6b1 WatchSource:0}: Error finding container 91b9e9985a17c8f97681ff21aaa60a26af8ea0a6a591b130398ed2a9c2f0e6b1: Status 404 returned error can't find the container with id 91b9e9985a17c8f97681ff21aaa60a26af8ea0a6a591b130398ed2a9c2f0e6b1
Nov 02 01:38:12.954999 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:12.954969    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6"]
Nov 02 01:38:12.957039 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:38:12.957013    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb145a144_84c0_43c6_b08b_c5c196efddf9.slice/crio-c21d1fe6a85b954690919f129a0571f9e7b9449a8c6e9be2840385bb9fa0e242 WatchSource:0}: Error finding container c21d1fe6a85b954690919f129a0571f9e7b9449a8c6e9be2840385bb9fa0e242: Status 404 returned error can't find the container with id c21d1fe6a85b954690919f129a0571f9e7b9449a8c6e9be2840385bb9fa0e242
Nov 02 01:38:13.602328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:13.602264    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6" event={"ID":"b145a144-84c0-43c6-b08b-c5c196efddf9","Type":"ContainerStarted","Data":"9da497a8309393231941ea3b21c7741a1c7a7bbaa0068ccc5f7c208c970b892c"}
Nov 02 01:38:13.602536 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:13.602351    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6" event={"ID":"b145a144-84c0-43c6-b08b-c5c196efddf9","Type":"ContainerStarted","Data":"c21d1fe6a85b954690919f129a0571f9e7b9449a8c6e9be2840385bb9fa0e242"}
Nov 02 01:38:13.602536 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:13.602375    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6"
Nov 02 01:38:13.603960 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:13.603937    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24" event={"ID":"5b302fb0-288a-4d1d-aece-d87bdf645f92","Type":"ContainerStarted","Data":"da409789a210797ce7fe0a508631a8a02e6494af1eaeae1c2793c48e65fd22e1"}
Nov 02 01:38:13.604101 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:13.603976    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24" event={"ID":"5b302fb0-288a-4d1d-aece-d87bdf645f92","Type":"ContainerStarted","Data":"91b9e9985a17c8f97681ff21aaa60a26af8ea0a6a591b130398ed2a9c2f0e6b1"}
Nov 02 01:38:13.604101 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:13.603997    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24"
Nov 02 01:38:13.719084 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:13.719029    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6" podStartSLOduration=2.719015069 podStartE2EDuration="2.719015069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:13.717016092 +0000 UTC m=+758.620698206" watchObservedRunningTime="2026-11-02 01:38:13.719015069 +0000 UTC m=+758.622697183"
Nov 02 01:38:13.848320 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:13.846928    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24" podStartSLOduration=1.846908747 podStartE2EDuration="1.846908747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:13.844627798 +0000 UTC m=+758.748309917" watchObservedRunningTime="2026-11-02 01:38:13.846908747 +0000 UTC m=+758.750590864"
Nov 02 01:38:14.534848 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:14.534821    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd"
Nov 02 01:38:16.970328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:16.969993    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc"]
Nov 02 01:38:16.977598 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:16.976944    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc"
Nov 02 01:38:17.032555 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:17.032525    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc"]
Nov 02 01:38:17.078489 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:17.078453    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-b71ba5\""
Nov 02 01:38:17.078489 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:17.078504    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-b71ba5-dockercfg-jgv4x\""
Nov 02 01:38:17.163794 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:17.163761    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hh6m\" (UniqueName: \"kubernetes.io/projected/2c6540a2-93f1-4867-ab73-bf3643ef90b5-kube-api-access-8hh6m\") pod \"payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc\" (UID: \"2c6540a2-93f1-4867-ab73-bf3643ef90b5\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc"
Nov 02 01:38:17.163996 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:17.163834    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c6540a2-93f1-4867-ab73-bf3643ef90b5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc\" (UID: \"2c6540a2-93f1-4867-ab73-bf3643ef90b5\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc"
Nov 02 01:38:17.265393 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:17.265262    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hh6m\" (UniqueName: \"kubernetes.io/projected/2c6540a2-93f1-4867-ab73-bf3643ef90b5-kube-api-access-8hh6m\") pod \"payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc\" (UID: \"2c6540a2-93f1-4867-ab73-bf3643ef90b5\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc"
Nov 02 01:38:17.265541 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:17.265453    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c6540a2-93f1-4867-ab73-bf3643ef90b5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc\" (UID: \"2c6540a2-93f1-4867-ab73-bf3643ef90b5\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc"
Nov 02 01:38:17.266137 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:17.266119    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c6540a2-93f1-4867-ab73-bf3643ef90b5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc\" (UID: \"2c6540a2-93f1-4867-ab73-bf3643ef90b5\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc"
Nov 02 01:38:17.326211 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:17.326177    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hh6m\" (UniqueName: \"kubernetes.io/projected/2c6540a2-93f1-4867-ab73-bf3643ef90b5-kube-api-access-8hh6m\") pod \"payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc\" (UID: \"2c6540a2-93f1-4867-ab73-bf3643ef90b5\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc"
Nov 02 01:38:17.612418 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:17.612384    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc"
Nov 02 01:38:17.988117 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:17.988093    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc"]
Nov 02 01:38:17.989989 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:38:17.989962    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c6540a2_93f1_4867_ab73_bf3643ef90b5.slice/crio-87105ffac8a797b13ee975e96841492d98444c377ee3e41ed8ff68cc00b45a78 WatchSource:0}: Error finding container 87105ffac8a797b13ee975e96841492d98444c377ee3e41ed8ff68cc00b45a78: Status 404 returned error can't find the container with id 87105ffac8a797b13ee975e96841492d98444c377ee3e41ed8ff68cc00b45a78
Nov 02 01:38:18.661006 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:18.660918    5639 generic.go:356] "Generic (PLEG): container finished" podID="8bf946ac-2c79-4b61-8890-703068477cdf" containerID="e95cf8fbdc7e541966df1c6e1eeb15cbc21cc51e2caacb56439a668f1d97211d" exitCode=0
Nov 02 01:38:18.661006 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:18.660994    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm" event={"ID":"8bf946ac-2c79-4b61-8890-703068477cdf","Type":"ContainerDied","Data":"e95cf8fbdc7e541966df1c6e1eeb15cbc21cc51e2caacb56439a668f1d97211d"}
Nov 02 01:38:18.661462 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:18.661433    5639 scope.go:122] "RemoveContainer" containerID="e95cf8fbdc7e541966df1c6e1eeb15cbc21cc51e2caacb56439a668f1d97211d"
Nov 02 01:38:18.662838 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:18.662817    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc" event={"ID":"2c6540a2-93f1-4867-ab73-bf3643ef90b5","Type":"ContainerStarted","Data":"23a6469a6741ad85f389aaf7d656c10df0ca3169706e73c49d181c1e53276b76"}
Nov 02 01:38:18.662947 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:18.662859    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc" event={"ID":"2c6540a2-93f1-4867-ab73-bf3643ef90b5","Type":"ContainerStarted","Data":"87105ffac8a797b13ee975e96841492d98444c377ee3e41ed8ff68cc00b45a78"}
Nov 02 01:38:18.662947 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:18.662874    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:38:18.662947 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:18.662921    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc"
Nov 02 01:38:18.662947 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:18.662939    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:38:18.663095 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:18.662974    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:38:18.947159 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:18.947057    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc" podStartSLOduration=2.9470442500000003 podStartE2EDuration="2.94704425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:18.946510503 +0000 UTC m=+763.850192619" watchObservedRunningTime="2026-11-02 01:38:18.94704425 +0000 UTC m=+763.850726364"
Nov 02 01:38:19.518903 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:19.514570    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:38:19.582537 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:19.582501    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2"
Nov 02 01:38:19.689232 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:19.689183    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm" event={"ID":"8bf946ac-2c79-4b61-8890-703068477cdf","Type":"ContainerStarted","Data":"9b916d457af7ad6a3530497528fa9fda585ef1052bcf4cffd69adcfcadbeb668"}
Nov 02 01:38:20.644786 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:20.644755    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:38:20.645449 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:20.645414    5639 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 02 01:38:20.645579 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:20.645480    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm" podUID="8bf946ac-2c79-4b61-8890-703068477cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 02 01:38:21.642837 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:21.642800    5639 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 02 01:38:21.643025 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:21.642855    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm" podUID="8bf946ac-2c79-4b61-8890-703068477cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 02 01:38:22.643513 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:22.643472    5639 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 02 01:38:22.643916 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:22.643574    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm" podUID="8bf946ac-2c79-4b61-8890-703068477cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 02 01:38:23.643293 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:23.643248    5639 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 02 01:38:23.643470 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:23.643323    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm" podUID="8bf946ac-2c79-4b61-8890-703068477cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 02 01:38:24.409314 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:24.408640    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:38:24.630556 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:24.630515    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24"
Nov 02 01:38:24.635552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:24.635524    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6"
Nov 02 01:38:24.642744 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:24.642715    5639 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 02 01:38:24.642901 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:24.642766    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm" podUID="8bf946ac-2c79-4b61-8890-703068477cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 02 01:38:25.647440 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:25.647411    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:38:25.648353 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:25.648330    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 01:38:27.080489 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:27.080454    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j"]
Nov 02 01:38:27.084046 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:27.084025    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j"
Nov 02 01:38:27.136348 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:27.136319    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-187478\""
Nov 02 01:38:27.181229 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:27.181178    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj7t7\" (UniqueName: \"kubernetes.io/projected/ef4d980a-8ce2-4357-aadc-d1839d056a61-kube-api-access-hj7t7\") pod \"payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j\" (UID: \"ef4d980a-8ce2-4357-aadc-d1839d056a61\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j"
Nov 02 01:38:27.181438 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:27.181255    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef4d980a-8ce2-4357-aadc-d1839d056a61-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j\" (UID: \"ef4d980a-8ce2-4357-aadc-d1839d056a61\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j"
Nov 02 01:38:27.189108 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:27.189073    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j"]
Nov 02 01:38:27.233901 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:27.233865    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-187478-564b56656f-xfg79"]
Nov 02 01:38:27.240242 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:27.240212    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-187478-564b56656f-xfg79"
Nov 02 01:38:27.282974 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:27.282925    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj7t7\" (UniqueName: \"kubernetes.io/projected/ef4d980a-8ce2-4357-aadc-d1839d056a61-kube-api-access-hj7t7\") pod \"payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j\" (UID: \"ef4d980a-8ce2-4357-aadc-d1839d056a61\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j"
Nov 02 01:38:27.283135 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:27.283003    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef4d980a-8ce2-4357-aadc-d1839d056a61-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j\" (UID: \"ef4d980a-8ce2-4357-aadc-d1839d056a61\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j"
Nov 02 01:38:27.283135 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:27.283058    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkt67\" (UniqueName: \"kubernetes.io/projected/67dc6edc-42f7-4e61-ad3c-38d3c3be4049-kube-api-access-vkt67\") pod \"payload-processing-e2e-worker-w4-187478-564b56656f-xfg79\" (UID: \"67dc6edc-42f7-4e61-ad3c-38d3c3be4049\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-187478-564b56656f-xfg79"
Nov 02 01:38:27.283238 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:27.283207    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67dc6edc-42f7-4e61-ad3c-38d3c3be4049-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-187478-564b56656f-xfg79\" (UID: \"67dc6edc-42f7-4e61-ad3c-38d3c3be4049\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-187478-564b56656f-xfg79"
Nov 02 01:38:27.284138 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:27.284114    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef4d980a-8ce2-4357-aadc-d1839d056a61-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j\" (UID: \"ef4d980a-8ce2-4357-aadc-d1839d056a61\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j"
Nov 02 01:38:27.295662 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:27.295614    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-187478-dockercfg-kkr5q\""
Nov 02 01:38:27.297362 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:27.297317    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-187478-564b56656f-xfg79"]
Nov 02 01:38:27.344904 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:27.344815    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj7t7\" (UniqueName: \"kubernetes.io/projected/ef4d980a-8ce2-4357-aadc-d1839d056a61-kube-api-access-hj7t7\") pod \"payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j\" (UID: \"ef4d980a-8ce2-4357-aadc-d1839d056a61\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j"
Nov 02 01:38:27.384610 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:27.384564    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkt67\" (UniqueName: \"kubernetes.io/projected/67dc6edc-42f7-4e61-ad3c-38d3c3be4049-kube-api-access-vkt67\") pod \"payload-processing-e2e-worker-w4-187478-564b56656f-xfg79\" (UID: \"67dc6edc-42f7-4e61-ad3c-38d3c3be4049\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-187478-564b56656f-xfg79"
Nov 02 01:38:27.384808 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:27.384705    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67dc6edc-42f7-4e61-ad3c-38d3c3be4049-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-187478-564b56656f-xfg79\" (UID: \"67dc6edc-42f7-4e61-ad3c-38d3c3be4049\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-187478-564b56656f-xfg79"
Nov 02 01:38:27.385479 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:27.385451    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67dc6edc-42f7-4e61-ad3c-38d3c3be4049-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-187478-564b56656f-xfg79\" (UID: \"67dc6edc-42f7-4e61-ad3c-38d3c3be4049\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-187478-564b56656f-xfg79"
Nov 02 01:38:27.402045 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:27.402012    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j"
Nov 02 01:38:27.425258 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:38:27.425229    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:38:27.446521 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:27.446485    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkt67\" (UniqueName: \"kubernetes.io/projected/67dc6edc-42f7-4e61-ad3c-38d3c3be4049-kube-api-access-vkt67\") pod \"payload-processing-e2e-worker-w4-187478-564b56656f-xfg79\" (UID: \"67dc6edc-42f7-4e61-ad3c-38d3c3be4049\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-187478-564b56656f-xfg79"
Nov 02 01:38:27.569368 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:27.569316    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-187478-564b56656f-xfg79"
Nov 02 01:38:27.790786 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:27.790758    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j"]
Nov 02 01:38:27.793298 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:38:27.793252    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef4d980a_8ce2_4357_aadc_d1839d056a61.slice/crio-57c252698c3e1ff3517cdd54f22d0d7cc9a4dd4fa0f607f0bbbbbd5acb22e5c5 WatchSource:0}: Error finding container 57c252698c3e1ff3517cdd54f22d0d7cc9a4dd4fa0f607f0bbbbbd5acb22e5c5: Status 404 returned error can't find the container with id 57c252698c3e1ff3517cdd54f22d0d7cc9a4dd4fa0f607f0bbbbbd5acb22e5c5
Nov 02 01:38:27.857546 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:27.857503    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j" event={"ID":"ef4d980a-8ce2-4357-aadc-d1839d056a61","Type":"ContainerStarted","Data":"57c252698c3e1ff3517cdd54f22d0d7cc9a4dd4fa0f607f0bbbbbd5acb22e5c5"}
Nov 02 01:38:27.947468 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:27.947436    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-187478-564b56656f-xfg79"]
Nov 02 01:38:27.949897 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:38:27.949875    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67dc6edc_42f7_4e61_ad3c_38d3c3be4049.slice/crio-f63d875de1514949074e77e1215dbb9ffe19243c3705e41504e7544a0ed598b5 WatchSource:0}: Error finding container f63d875de1514949074e77e1215dbb9ffe19243c3705e41504e7544a0ed598b5: Status 404 returned error can't find the container with id f63d875de1514949074e77e1215dbb9ffe19243c3705e41504e7544a0ed598b5
Nov 02 01:38:28.867067 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:28.867017    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-187478-564b56656f-xfg79" event={"ID":"67dc6edc-42f7-4e61-ad3c-38d3c3be4049","Type":"ContainerStarted","Data":"f7aa833139a2978502ef2d38fba0f4721bb2c8f1d3447b716a82de85e122b721"}
Nov 02 01:38:28.867696 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:28.867088    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-187478-564b56656f-xfg79" event={"ID":"67dc6edc-42f7-4e61-ad3c-38d3c3be4049","Type":"ContainerStarted","Data":"f63d875de1514949074e77e1215dbb9ffe19243c3705e41504e7544a0ed598b5"}
Nov 02 01:38:28.867696 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:28.867140    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-187478-564b56656f-xfg79"
Nov 02 01:38:28.868902 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:28.868877    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j" event={"ID":"ef4d980a-8ce2-4357-aadc-d1839d056a61","Type":"ContainerStarted","Data":"42927e6f4c4a09dab66ce743c20ef756601aafcfaab15329876057edb9ac9fc0"}
Nov 02 01:38:28.869033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:28.868939    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j"
Nov 02 01:38:29.107135 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:29.107061    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j" podStartSLOduration=2.107041502 podStartE2EDuration="2.107041502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:29.101397719 +0000 UTC m=+774.005079835" watchObservedRunningTime="2026-11-02 01:38:29.107041502 +0000 UTC m=+774.010723617"
Nov 02 01:38:29.107475 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:29.107430    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-187478-564b56656f-xfg79" podStartSLOduration=2.107419655 podStartE2EDuration="2.107419655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:28.985803343 +0000 UTC m=+773.889485457" watchObservedRunningTime="2026-11-02 01:38:29.107419655 +0000 UTC m=+774.011101770"
Nov 02 01:38:29.692427 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:29.692399    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc"
Nov 02 01:38:29.877879 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:29.877850    5639 generic.go:356] "Generic (PLEG): container finished" podID="2da5e48a-b3fd-4b3a-97d9-505aea24dc94" containerID="fd7ec3605ebe009a0d9fb2a852381c7e698fc9de83f5d8bfd0f56ccc28447ef7" exitCode=0
Nov 02 01:38:29.878372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:29.877919    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq" event={"ID":"2da5e48a-b3fd-4b3a-97d9-505aea24dc94","Type":"ContainerDied","Data":"fd7ec3605ebe009a0d9fb2a852381c7e698fc9de83f5d8bfd0f56ccc28447ef7"}
Nov 02 01:38:29.878437 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:29.878387    5639 scope.go:122] "RemoveContainer" containerID="fd7ec3605ebe009a0d9fb2a852381c7e698fc9de83f5d8bfd0f56ccc28447ef7"
Nov 02 01:38:29.879731 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:29.879673    5639 generic.go:356] "Generic (PLEG): container finished" podID="eb685866-3bb8-43ae-85f9-17cea849a9f8" containerID="8d940a2ad7728f0ce218a8285737b15ad93fdda1762f17cfa23c346b8ade176c" exitCode=0
Nov 02 01:38:29.879817 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:29.879746    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp" event={"ID":"eb685866-3bb8-43ae-85f9-17cea849a9f8","Type":"ContainerDied","Data":"8d940a2ad7728f0ce218a8285737b15ad93fdda1762f17cfa23c346b8ade176c"}
Nov 02 01:38:29.879956 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:29.879925    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:38:29.880023 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:29.880008    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:38:29.880074 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:29.880059    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:38:29.880496 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:29.880480    5639 scope.go:122] "RemoveContainer" containerID="8d940a2ad7728f0ce218a8285737b15ad93fdda1762f17cfa23c346b8ade176c"
Nov 02 01:38:29.881613 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:29.881587    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:38:29.881724 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:29.881648    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:38:29.881724 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:29.881680    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:38:30.817304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:30.810598    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:38:30.922368 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:30.922327    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq" event={"ID":"2da5e48a-b3fd-4b3a-97d9-505aea24dc94","Type":"ContainerStarted","Data":"65698364ceb649fb3159bbb0894b582246cc6b83cad9652e1d5b40cd7790cb41"}
Nov 02 01:38:30.925030 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:30.925002    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp" event={"ID":"eb685866-3bb8-43ae-85f9-17cea849a9f8","Type":"ContainerStarted","Data":"acd0899b4ff5fccb1881344f98a75bbcac40f58045b5de9b6edec43db513774d"}
Nov 02 01:38:31.009303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:31.008685    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:38:31.009303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:31.008821    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 02 01:38:31.009303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:31.008866    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp" podUID="eb685866-3bb8-43ae-85f9-17cea849a9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 02 01:38:31.597307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:31.596721    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:38:31.597307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:31.596846    5639 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Nov 02 01:38:31.597307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:31.596890    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq" podUID="2da5e48a-b3fd-4b3a-97d9-505aea24dc94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Nov 02 01:38:32.086374 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:32.080327    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 02 01:38:32.086374 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:32.080425    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp" podUID="eb685866-3bb8-43ae-85f9-17cea849a9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 02 01:38:32.596304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:32.595861    5639 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Nov 02 01:38:32.596304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:32.595940    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq" podUID="2da5e48a-b3fd-4b3a-97d9-505aea24dc94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Nov 02 01:38:33.014382 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:33.014054    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 02 01:38:33.014382 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:33.014122    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp" podUID="eb685866-3bb8-43ae-85f9-17cea849a9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 02 01:38:33.592746 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:33.592707    5639 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Nov 02 01:38:33.593386 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:33.593351    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq" podUID="2da5e48a-b3fd-4b3a-97d9-505aea24dc94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Nov 02 01:38:34.008793 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:34.008702    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 02 01:38:34.009068 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:34.009038    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp" podUID="eb685866-3bb8-43ae-85f9-17cea849a9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 02 01:38:34.607308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:34.595679    5639 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Nov 02 01:38:34.607308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:34.595740    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq" podUID="2da5e48a-b3fd-4b3a-97d9-505aea24dc94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Nov 02 01:38:35.018372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:35.017735    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 02 01:38:35.018372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:35.017814    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp" podUID="eb685866-3bb8-43ae-85f9-17cea849a9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 02 01:38:35.595309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:35.592696    5639 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Nov 02 01:38:35.595309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:35.592759    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq" podUID="2da5e48a-b3fd-4b3a-97d9-505aea24dc94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Nov 02 01:38:36.009800 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:36.009702    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 02 01:38:36.010432 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:36.010398    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp" podUID="eb685866-3bb8-43ae-85f9-17cea849a9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 02 01:38:36.596770 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:36.596742    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:38:36.597063 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:36.597043    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:38:36.597757 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:36.597739    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 01:38:37.012941 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:37.012856    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:38:37.013431 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:37.013225    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:38:37.013955 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:37.013935    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 01:38:38.434487 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:38.434441    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:38:38.435045 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:38.434526    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2" podUID="c4dec6db-d72f-45ce-802c-f54bb7f57e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:38:39.902563 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:39.902535    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j"
Nov 02 01:38:39.904225 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:39.904199    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-187478-564b56656f-xfg79"
Nov 02 01:38:40.087653 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:40.087615    5639 generic.go:356] "Generic (PLEG): container finished" podID="c4dec6db-d72f-45ce-802c-f54bb7f57e5c" containerID="686c26bc415f61deefbc110374f8064e90e1eb57966696875b12338d48775fbd" exitCode=0
Nov 02 01:38:40.088094 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:40.087690    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2" event={"ID":"c4dec6db-d72f-45ce-802c-f54bb7f57e5c","Type":"ContainerDied","Data":"686c26bc415f61deefbc110374f8064e90e1eb57966696875b12338d48775fbd"}
Nov 02 01:38:40.089089 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:40.088642    5639 scope.go:122] "RemoveContainer" containerID="686c26bc415f61deefbc110374f8064e90e1eb57966696875b12338d48775fbd"
Nov 02 01:38:40.090478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:40.090443    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:38:40.090565 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:40.090528    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:38:40.090604 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:40.090576    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:38:40.090604 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:40.090584    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:38:41.095926 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:41.095893    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2" event={"ID":"c4dec6db-d72f-45ce-802c-f54bb7f57e5c","Type":"ContainerStarted","Data":"4e7d9c762d5d8ba4b4170e6817722b48b66635bad48a192409959689c9bc9469"}
Nov 02 01:38:41.390374 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:41.389697    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:38:41.390374 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:41.389823    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 02 01:38:41.390374 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:41.389864    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2" podUID="c4dec6db-d72f-45ce-802c-f54bb7f57e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 02 01:38:42.384746 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:42.384704    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 02 01:38:42.385265 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:42.384763    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2" podUID="c4dec6db-d72f-45ce-802c-f54bb7f57e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 02 01:38:43.384814 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:43.384773    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 02 01:38:43.385254 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:43.384847    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2" podUID="c4dec6db-d72f-45ce-802c-f54bb7f57e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 02 01:38:44.384655 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:44.384624    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 02 01:38:44.384862 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:44.384679    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2" podUID="c4dec6db-d72f-45ce-802c-f54bb7f57e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 02 01:38:45.384772 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:45.384738    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 02 01:38:45.385091 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:45.385055    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2" podUID="c4dec6db-d72f-45ce-802c-f54bb7f57e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 02 01:38:46.388804 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:46.388773    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:38:46.389317 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:46.389158    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:38:46.389850 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:46.389833    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:38:57.425217 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:38:57.425180    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:38:59.378125 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:38:59.378096    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:39:09.004106 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.004066    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"]
Nov 02 01:39:09.009409 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.009386    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:39:09.065784 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.065751    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-36a1e0\"/\"openshift-service-ca.crt\""
Nov 02 01:39:09.114798 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.114774    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-36a1e0\"/\"kube-root-ca.crt\""
Nov 02 01:39:09.114996 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.114984    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-36a1e0\"/\"simulator-w2-36a1e0-kserve-self-signed-certs\""
Nov 02 01:39:09.115251 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.115232    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-36a1e0\"/\"default-dockercfg-t2p55\""
Nov 02 01:39:09.116776 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.116758    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"]
Nov 02 01:39:09.174213 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.174112    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps827\" (UniqueName: \"kubernetes.io/projected/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-kube-api-access-ps827\") pod \"simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr\" (UID: \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\") " pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:39:09.174563 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.174539    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-tmp-dir\") pod \"simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr\" (UID: \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\") " pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:39:09.174797 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.174778    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-dshm\") pod \"simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr\" (UID: \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\") " pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:39:09.175032 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.175017    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-home\") pod \"simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr\" (UID: \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\") " pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:39:09.175328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.175266    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-model-cache\") pod \"simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr\" (UID: \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\") " pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:39:09.175470 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.175455    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-tls-certs\") pod \"simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr\" (UID: \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\") " pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:39:09.276585 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.276484    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-home\") pod \"simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr\" (UID: \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\") " pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:39:09.276775 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.276586    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-model-cache\") pod \"simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr\" (UID: \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\") " pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:39:09.276775 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.276614    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-tls-certs\") pod \"simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr\" (UID: \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\") " pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:39:09.276775 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.276636    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps827\" (UniqueName: \"kubernetes.io/projected/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-kube-api-access-ps827\") pod \"simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr\" (UID: \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\") " pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:39:09.276775 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.276700    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-tmp-dir\") pod \"simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr\" (UID: \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\") " pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:39:09.276775 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.276722    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-dshm\") pod \"simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr\" (UID: \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\") " pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:39:09.277049 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.276926    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-home\") pod \"simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr\" (UID: \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\") " pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:39:09.277049 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.276980    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-model-cache\") pod \"simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr\" (UID: \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\") " pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:39:09.277159 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.277051    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-tmp-dir\") pod \"simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr\" (UID: \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\") " pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:39:09.279236 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.279206    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-dshm\") pod \"simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr\" (UID: \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\") " pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:39:09.279408 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.279392    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-tls-certs\") pod \"simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr\" (UID: \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\") " pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:39:09.339945 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.339917    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps827\" (UniqueName: \"kubernetes.io/projected/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-kube-api-access-ps827\") pod \"simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr\" (UID: \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\") " pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:39:09.350823 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.350804    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:39:09.721836 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:09.721679    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"]
Nov 02 01:39:09.724228 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:39:09.724148    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43bdf2ad_6e64_46da_ad9f_4b9dc775290a.slice/crio-cd039704ac51723a29a50cf45bd6466a074567b49cea925c594416ef27ddecdf WatchSource:0}: Error finding container cd039704ac51723a29a50cf45bd6466a074567b49cea925c594416ef27ddecdf: Status 404 returned error can't find the container with id cd039704ac51723a29a50cf45bd6466a074567b49cea925c594416ef27ddecdf
Nov 02 01:39:10.264135 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:10.264106    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" event={"ID":"43bdf2ad-6e64-46da-ad9f-4b9dc775290a","Type":"ContainerStarted","Data":"227cdb9dc495fe33b66b2dbabe9dccf7cd2c3698beda12c383961a378665f37f"}
Nov 02 01:39:10.264534 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:10.264146    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" event={"ID":"43bdf2ad-6e64-46da-ad9f-4b9dc775290a","Type":"ContainerStarted","Data":"cd039704ac51723a29a50cf45bd6466a074567b49cea925c594416ef27ddecdf"}
Nov 02 01:39:10.351708 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:10.351681    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:39:10.384177 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:10.384116    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podStartSLOduration=2.384102828 podStartE2EDuration="2.384102828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:39:10.381836208 +0000 UTC m=+815.285518321" watchObservedRunningTime="2026-11-02 01:39:10.384102828 +0000 UTC m=+815.287784947"
Nov 02 01:39:13.195587 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:13.195549    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:39:19.351686 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:19.351608    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:39:19.364212 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:19.364184    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:39:20.346085 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:20.346057    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:39:22.502142 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:22.502109    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"]
Nov 02 01:39:22.505830 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:22.505814    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:39:22.560684 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:22.560658    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-88e76d\"/\"default-dockercfg-2rrmt\""
Nov 02 01:39:22.610156 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:22.610130    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-88e76d\"/\"simulator-w3-88e76d-kserve-self-signed-certs\""
Nov 02 01:39:22.610537 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:22.610521    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-88e76d\"/\"kube-root-ca.crt\""
Nov 02 01:39:22.610660 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:22.610538    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-88e76d\"/\"openshift-service-ca.crt\""
Nov 02 01:39:22.612791 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:22.612770    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"]
Nov 02 01:39:22.648815 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:22.648777    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbwnb\" (UniqueName: \"kubernetes.io/projected/dfbf279b-0423-45be-a91c-f35427015145-kube-api-access-gbwnb\") pod \"simulator-w3-88e76d-kserve-5ccd676f8d-66spz\" (UID: \"dfbf279b-0423-45be-a91c-f35427015145\") " pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:39:22.649183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:22.649153    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-model-cache\") pod \"simulator-w3-88e76d-kserve-5ccd676f8d-66spz\" (UID: \"dfbf279b-0423-45be-a91c-f35427015145\") " pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:39:22.649465 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:22.649238    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfbf279b-0423-45be-a91c-f35427015145-tls-certs\") pod \"simulator-w3-88e76d-kserve-5ccd676f8d-66spz\" (UID: \"dfbf279b-0423-45be-a91c-f35427015145\") " pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:39:22.649465 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:22.649315    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-home\") pod \"simulator-w3-88e76d-kserve-5ccd676f8d-66spz\" (UID: \"dfbf279b-0423-45be-a91c-f35427015145\") " pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:39:22.649465 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:22.649392    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-tmp-dir\") pod \"simulator-w3-88e76d-kserve-5ccd676f8d-66spz\" (UID: \"dfbf279b-0423-45be-a91c-f35427015145\") " pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:39:22.649465 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:22.649418    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-dshm\") pod \"simulator-w3-88e76d-kserve-5ccd676f8d-66spz\" (UID: \"dfbf279b-0423-45be-a91c-f35427015145\") " pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:39:22.750463 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:22.750425    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-home\") pod \"simulator-w3-88e76d-kserve-5ccd676f8d-66spz\" (UID: \"dfbf279b-0423-45be-a91c-f35427015145\") " pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:39:22.750463 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:22.750468    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-tmp-dir\") pod \"simulator-w3-88e76d-kserve-5ccd676f8d-66spz\" (UID: \"dfbf279b-0423-45be-a91c-f35427015145\") " pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:39:22.750716 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:22.750484    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-dshm\") pod \"simulator-w3-88e76d-kserve-5ccd676f8d-66spz\" (UID: \"dfbf279b-0423-45be-a91c-f35427015145\") " pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:39:22.750716 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:22.750548    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbwnb\" (UniqueName: \"kubernetes.io/projected/dfbf279b-0423-45be-a91c-f35427015145-kube-api-access-gbwnb\") pod \"simulator-w3-88e76d-kserve-5ccd676f8d-66spz\" (UID: \"dfbf279b-0423-45be-a91c-f35427015145\") " pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:39:22.750716 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:22.750616    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-model-cache\") pod \"simulator-w3-88e76d-kserve-5ccd676f8d-66spz\" (UID: \"dfbf279b-0423-45be-a91c-f35427015145\") " pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:39:22.750716 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:22.750652    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfbf279b-0423-45be-a91c-f35427015145-tls-certs\") pod \"simulator-w3-88e76d-kserve-5ccd676f8d-66spz\" (UID: \"dfbf279b-0423-45be-a91c-f35427015145\") " pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:39:22.750935 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:22.750883    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-tmp-dir\") pod \"simulator-w3-88e76d-kserve-5ccd676f8d-66spz\" (UID: \"dfbf279b-0423-45be-a91c-f35427015145\") " pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:39:22.750935 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:22.750915    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-home\") pod \"simulator-w3-88e76d-kserve-5ccd676f8d-66spz\" (UID: \"dfbf279b-0423-45be-a91c-f35427015145\") " pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:39:22.751030 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:22.750990    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-model-cache\") pod \"simulator-w3-88e76d-kserve-5ccd676f8d-66spz\" (UID: \"dfbf279b-0423-45be-a91c-f35427015145\") " pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:39:22.753004 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:22.752955    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-dshm\") pod \"simulator-w3-88e76d-kserve-5ccd676f8d-66spz\" (UID: \"dfbf279b-0423-45be-a91c-f35427015145\") " pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:39:22.753135 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:22.753117    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfbf279b-0423-45be-a91c-f35427015145-tls-certs\") pod \"simulator-w3-88e76d-kserve-5ccd676f8d-66spz\" (UID: \"dfbf279b-0423-45be-a91c-f35427015145\") " pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:39:22.842172 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:22.842138    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbwnb\" (UniqueName: \"kubernetes.io/projected/dfbf279b-0423-45be-a91c-f35427015145-kube-api-access-gbwnb\") pod \"simulator-w3-88e76d-kserve-5ccd676f8d-66spz\" (UID: \"dfbf279b-0423-45be-a91c-f35427015145\") " pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:39:23.118623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:23.118589    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:39:23.494202 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:23.494171    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"]
Nov 02 01:39:23.494951 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:39:23.494923    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddfbf279b_0423_45be_a91c_f35427015145.slice/crio-9413d2faf8ca66df91275f1bb045f29a2bbbae629f7ecf5675bc965bf8de70f2 WatchSource:0}: Error finding container 9413d2faf8ca66df91275f1bb045f29a2bbbae629f7ecf5675bc965bf8de70f2: Status 404 returned error can't find the container with id 9413d2faf8ca66df91275f1bb045f29a2bbbae629f7ecf5675bc965bf8de70f2
Nov 02 01:39:24.359303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:24.359239    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" event={"ID":"dfbf279b-0423-45be-a91c-f35427015145","Type":"ContainerStarted","Data":"41abe20bbd71632a17d73c6da7b7538f458b931be2b9bef48b731a29705654e8"}
Nov 02 01:39:24.359711 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:24.359318    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" event={"ID":"dfbf279b-0423-45be-a91c-f35427015145","Type":"ContainerStarted","Data":"9413d2faf8ca66df91275f1bb045f29a2bbbae629f7ecf5675bc965bf8de70f2"}
Nov 02 01:39:24.477673 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:24.477625    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podStartSLOduration=2.477610443 podStartE2EDuration="2.477610443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:39:24.476258295 +0000 UTC m=+829.379940432" watchObservedRunningTime="2026-11-02 01:39:24.477610443 +0000 UTC m=+829.381292549"
Nov 02 01:39:25.118823 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:25.118791    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:39:27.421418 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:39:27.421384    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:39:31.199655 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:31.199619    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:39:33.119235 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:33.119199    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:39:33.132326 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:33.132264    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:39:33.416533 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:33.416463    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:39:46.968135 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:46.968100    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:39:55.011060 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.011026    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"]
Nov 02 01:39:55.014734 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.014718    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:39:55.116372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.116337    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-187478\"/\"openshift-service-ca.crt\""
Nov 02 01:39:55.116587 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.116427    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-187478\"/\"default-dockercfg-qnkcp\""
Nov 02 01:39:55.116587 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.116514    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-187478\"/\"kube-root-ca.crt\""
Nov 02 01:39:55.116689 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.116601    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-187478\"/\"simulator-w4-187478-kserve-self-signed-certs\""
Nov 02 01:39:55.119805 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.119779    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"]
Nov 02 01:39:55.134049 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.134022    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-model-cache\") pod \"simulator-w4-187478-kserve-c4b76d8d-cr66c\" (UID: \"7d97942c-a6d9-4272-ac79-51a0986061b7\") " pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:39:55.134156 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.134068    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-dshm\") pod \"simulator-w4-187478-kserve-c4b76d8d-cr66c\" (UID: \"7d97942c-a6d9-4272-ac79-51a0986061b7\") " pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:39:55.134207 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.134190    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d97942c-a6d9-4272-ac79-51a0986061b7-tls-certs\") pod \"simulator-w4-187478-kserve-c4b76d8d-cr66c\" (UID: \"7d97942c-a6d9-4272-ac79-51a0986061b7\") " pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:39:55.134372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.134353    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-home\") pod \"simulator-w4-187478-kserve-c4b76d8d-cr66c\" (UID: \"7d97942c-a6d9-4272-ac79-51a0986061b7\") " pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:39:55.134421 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.134406    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftjxl\" (UniqueName: \"kubernetes.io/projected/7d97942c-a6d9-4272-ac79-51a0986061b7-kube-api-access-ftjxl\") pod \"simulator-w4-187478-kserve-c4b76d8d-cr66c\" (UID: \"7d97942c-a6d9-4272-ac79-51a0986061b7\") " pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:39:55.134478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.134467    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-tmp-dir\") pod \"simulator-w4-187478-kserve-c4b76d8d-cr66c\" (UID: \"7d97942c-a6d9-4272-ac79-51a0986061b7\") " pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:39:55.236056 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.236021    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftjxl\" (UniqueName: \"kubernetes.io/projected/7d97942c-a6d9-4272-ac79-51a0986061b7-kube-api-access-ftjxl\") pod \"simulator-w4-187478-kserve-c4b76d8d-cr66c\" (UID: \"7d97942c-a6d9-4272-ac79-51a0986061b7\") " pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:39:55.236056 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.236064    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-tmp-dir\") pod \"simulator-w4-187478-kserve-c4b76d8d-cr66c\" (UID: \"7d97942c-a6d9-4272-ac79-51a0986061b7\") " pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:39:55.236364 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.236087    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-model-cache\") pod \"simulator-w4-187478-kserve-c4b76d8d-cr66c\" (UID: \"7d97942c-a6d9-4272-ac79-51a0986061b7\") " pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:39:55.236364 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.236104    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-dshm\") pod \"simulator-w4-187478-kserve-c4b76d8d-cr66c\" (UID: \"7d97942c-a6d9-4272-ac79-51a0986061b7\") " pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:39:55.236364 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.236158    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d97942c-a6d9-4272-ac79-51a0986061b7-tls-certs\") pod \"simulator-w4-187478-kserve-c4b76d8d-cr66c\" (UID: \"7d97942c-a6d9-4272-ac79-51a0986061b7\") " pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:39:55.236364 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.236246    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-home\") pod \"simulator-w4-187478-kserve-c4b76d8d-cr66c\" (UID: \"7d97942c-a6d9-4272-ac79-51a0986061b7\") " pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:39:55.236608 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.236560    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-tmp-dir\") pod \"simulator-w4-187478-kserve-c4b76d8d-cr66c\" (UID: \"7d97942c-a6d9-4272-ac79-51a0986061b7\") " pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:39:55.236608 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.236584    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-model-cache\") pod \"simulator-w4-187478-kserve-c4b76d8d-cr66c\" (UID: \"7d97942c-a6d9-4272-ac79-51a0986061b7\") " pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:39:55.236688 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.236623    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-home\") pod \"simulator-w4-187478-kserve-c4b76d8d-cr66c\" (UID: \"7d97942c-a6d9-4272-ac79-51a0986061b7\") " pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:39:55.238600 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.238572    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-dshm\") pod \"simulator-w4-187478-kserve-c4b76d8d-cr66c\" (UID: \"7d97942c-a6d9-4272-ac79-51a0986061b7\") " pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:39:55.238769 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.238751    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d97942c-a6d9-4272-ac79-51a0986061b7-tls-certs\") pod \"simulator-w4-187478-kserve-c4b76d8d-cr66c\" (UID: \"7d97942c-a6d9-4272-ac79-51a0986061b7\") " pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:39:55.296899 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.296828    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftjxl\" (UniqueName: \"kubernetes.io/projected/7d97942c-a6d9-4272-ac79-51a0986061b7-kube-api-access-ftjxl\") pod \"simulator-w4-187478-kserve-c4b76d8d-cr66c\" (UID: \"7d97942c-a6d9-4272-ac79-51a0986061b7\") " pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:39:55.326658 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.326624    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:39:55.714905 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:55.714883    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"]
Nov 02 01:39:55.717146 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:39:55.717119    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d97942c_a6d9_4272_ac79_51a0986061b7.slice/crio-c1b4d976aaf83d14771b3810f56015689daacad54928b3ee7775867963d94eea WatchSource:0}: Error finding container c1b4d976aaf83d14771b3810f56015689daacad54928b3ee7775867963d94eea: Status 404 returned error can't find the container with id c1b4d976aaf83d14771b3810f56015689daacad54928b3ee7775867963d94eea
Nov 02 01:39:56.543382 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:56.543344    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" event={"ID":"7d97942c-a6d9-4272-ac79-51a0986061b7","Type":"ContainerStarted","Data":"2dccefb602dc3bde210391df8e1a13c0d9d5c0a5bdc36c2bff8b7131e0d98edc"}
Nov 02 01:39:56.543382 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:56.543386    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" event={"ID":"7d97942c-a6d9-4272-ac79-51a0986061b7","Type":"ContainerStarted","Data":"c1b4d976aaf83d14771b3810f56015689daacad54928b3ee7775867963d94eea"}
Nov 02 01:39:56.660988 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:56.660927    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podStartSLOduration=2.66091225 podStartE2EDuration="2.66091225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:39:56.6590514 +0000 UTC m=+861.562733517" watchObservedRunningTime="2026-11-02 01:39:56.66091225 +0000 UTC m=+861.564594394"
Nov 02 01:39:57.327636 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:57.327603    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:39:57.420423 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:39:57.420391    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:39:58.642143 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:58.642106    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"]
Nov 02 01:39:58.646178 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:58.646158    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:39:58.751333 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:58.751267    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5888dc\"/\"openshift-service-ca.crt\""
Nov 02 01:39:58.751333 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:58.751267    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5888dc\"/\"kube-root-ca.crt\""
Nov 02 01:39:58.751556 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:58.751344    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5888dc\"/\"default-dockercfg-mxnhx\""
Nov 02 01:39:58.751556 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:58.751349    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5888dc\"/\"simulator-w6-5888dc-kserve-self-signed-certs\""
Nov 02 01:39:58.753597 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:58.753574    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"]
Nov 02 01:39:58.772998 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:58.772975    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-tls-certs\") pod \"simulator-w6-5888dc-kserve-749df46745-rvzn6\" (UID: \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\") " pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:39:58.773078 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:58.773029    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-dshm\") pod \"simulator-w6-5888dc-kserve-749df46745-rvzn6\" (UID: \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\") " pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:39:58.773078 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:58.773045    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-tmp-dir\") pod \"simulator-w6-5888dc-kserve-749df46745-rvzn6\" (UID: \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\") " pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:39:58.773179 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:58.773165    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-model-cache\") pod \"simulator-w6-5888dc-kserve-749df46745-rvzn6\" (UID: \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\") " pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:39:58.773219 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:58.773199    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxftz\" (UniqueName: \"kubernetes.io/projected/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-kube-api-access-bxftz\") pod \"simulator-w6-5888dc-kserve-749df46745-rvzn6\" (UID: \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\") " pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:39:58.773252 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:58.773236    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-home\") pod \"simulator-w6-5888dc-kserve-749df46745-rvzn6\" (UID: \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\") " pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:39:58.874333 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:58.874264    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-model-cache\") pod \"simulator-w6-5888dc-kserve-749df46745-rvzn6\" (UID: \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\") " pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:39:58.874478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:58.874372    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxftz\" (UniqueName: \"kubernetes.io/projected/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-kube-api-access-bxftz\") pod \"simulator-w6-5888dc-kserve-749df46745-rvzn6\" (UID: \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\") " pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:39:58.874478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:58.874418    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-home\") pod \"simulator-w6-5888dc-kserve-749df46745-rvzn6\" (UID: \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\") " pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:39:58.874587 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:58.874518    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-tls-certs\") pod \"simulator-w6-5888dc-kserve-749df46745-rvzn6\" (UID: \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\") " pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:39:58.874640 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:58.874592    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-dshm\") pod \"simulator-w6-5888dc-kserve-749df46745-rvzn6\" (UID: \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\") " pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:39:58.874640 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:58.874613    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-tmp-dir\") pod \"simulator-w6-5888dc-kserve-749df46745-rvzn6\" (UID: \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\") " pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:39:58.874766 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:58.874744    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-model-cache\") pod \"simulator-w6-5888dc-kserve-749df46745-rvzn6\" (UID: \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\") " pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:39:58.874834 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:58.874775    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-home\") pod \"simulator-w6-5888dc-kserve-749df46745-rvzn6\" (UID: \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\") " pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:39:58.874957 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:58.874937    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-tmp-dir\") pod \"simulator-w6-5888dc-kserve-749df46745-rvzn6\" (UID: \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\") " pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:39:58.877010 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:58.876977    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-dshm\") pod \"simulator-w6-5888dc-kserve-749df46745-rvzn6\" (UID: \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\") " pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:39:58.877263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:58.877244    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-tls-certs\") pod \"simulator-w6-5888dc-kserve-749df46745-rvzn6\" (UID: \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\") " pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:39:58.935708 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:58.935642    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxftz\" (UniqueName: \"kubernetes.io/projected/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-kube-api-access-bxftz\") pod \"simulator-w6-5888dc-kserve-749df46745-rvzn6\" (UID: \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\") " pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:39:58.958939 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:58.958914    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:39:59.333662 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:59.333639    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"]
Nov 02 01:39:59.335810 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:39:59.335783    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f4b634d_f8b6_4faf_911b_12b3b7990fe1.slice/crio-25644ce7742a511d0fa5e0c0c27bc85e3d89e29aa0c199878c1e363a1065aa32 WatchSource:0}: Error finding container 25644ce7742a511d0fa5e0c0c27bc85e3d89e29aa0c199878c1e363a1065aa32: Status 404 returned error can't find the container with id 25644ce7742a511d0fa5e0c0c27bc85e3d89e29aa0c199878c1e363a1065aa32
Nov 02 01:39:59.561583 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:59.561551    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" event={"ID":"5f4b634d-f8b6-4faf-911b-12b3b7990fe1","Type":"ContainerStarted","Data":"13427df22a25f684b69773ba0e4d9c76bd848aa0d1f0e56dfbb3f3b7298e5d37"}
Nov 02 01:39:59.561583 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:59.561587    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" event={"ID":"5f4b634d-f8b6-4faf-911b-12b3b7990fe1","Type":"ContainerStarted","Data":"25644ce7742a511d0fa5e0c0c27bc85e3d89e29aa0c199878c1e363a1065aa32"}
Nov 02 01:39:59.678317 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:59.678173    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podStartSLOduration=1.6781583740000001 podStartE2EDuration="1.678158374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:39:59.67778563 +0000 UTC m=+864.581467745" watchObservedRunningTime="2026-11-02 01:39:59.678158374 +0000 UTC m=+864.581840488"
Nov 02 01:39:59.959858 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:39:59.959782    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:40:03.583833 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:03.583799    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:40:05.327570 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:05.327540    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:40:05.340240 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:05.340215    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:40:05.604639 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:05.604551    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:40:08.959247 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:08.959215    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:40:08.972096 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:08.972066    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:40:09.627900 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:09.627869    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:40:26.840167 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:26.840131    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:40:27.508115 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:40:27.508057    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:40:47.589387 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:47.589308    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:40:51.649612 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:51.649576    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"]
Nov 02 01:40:51.653221 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:51.653205    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:40:51.754723 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:51.754687    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-88e76d\"/\"premium-w3-88e76d-kserve-self-signed-certs\""
Nov 02 01:40:51.757032 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:51.757006    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"]
Nov 02 01:40:51.780957 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:51.780905    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-model-cache\") pod \"premium-w3-88e76d-kserve-6b957679cc-gjgjw\" (UID: \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\") " pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:40:51.780957 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:51.780948    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-tmp-dir\") pod \"premium-w3-88e76d-kserve-6b957679cc-gjgjw\" (UID: \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\") " pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:40:51.781233 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:51.781007    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-dshm\") pod \"premium-w3-88e76d-kserve-6b957679cc-gjgjw\" (UID: \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\") " pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:40:51.781233 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:51.781088    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b10e1d24-9716-4ff6-89c2-e59452b79a8b-tls-certs\") pod \"premium-w3-88e76d-kserve-6b957679cc-gjgjw\" (UID: \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\") " pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:40:51.781233 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:51.781126    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-home\") pod \"premium-w3-88e76d-kserve-6b957679cc-gjgjw\" (UID: \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\") " pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:40:51.781233 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:51.781168    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npm5x\" (UniqueName: \"kubernetes.io/projected/b10e1d24-9716-4ff6-89c2-e59452b79a8b-kube-api-access-npm5x\") pod \"premium-w3-88e76d-kserve-6b957679cc-gjgjw\" (UID: \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\") " pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:40:51.882487 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:51.882458    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npm5x\" (UniqueName: \"kubernetes.io/projected/b10e1d24-9716-4ff6-89c2-e59452b79a8b-kube-api-access-npm5x\") pod \"premium-w3-88e76d-kserve-6b957679cc-gjgjw\" (UID: \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\") " pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:40:51.882679 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:51.882545    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-model-cache\") pod \"premium-w3-88e76d-kserve-6b957679cc-gjgjw\" (UID: \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\") " pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:40:51.882679 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:51.882564    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-tmp-dir\") pod \"premium-w3-88e76d-kserve-6b957679cc-gjgjw\" (UID: \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\") " pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:40:51.882679 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:51.882598    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-dshm\") pod \"premium-w3-88e76d-kserve-6b957679cc-gjgjw\" (UID: \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\") " pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:40:51.882679 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:51.882649    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b10e1d24-9716-4ff6-89c2-e59452b79a8b-tls-certs\") pod \"premium-w3-88e76d-kserve-6b957679cc-gjgjw\" (UID: \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\") " pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:40:51.882679 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:51.882664    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-home\") pod \"premium-w3-88e76d-kserve-6b957679cc-gjgjw\" (UID: \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\") " pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:40:51.883004 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:51.882986    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-model-cache\") pod \"premium-w3-88e76d-kserve-6b957679cc-gjgjw\" (UID: \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\") " pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:40:51.883058 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:51.883033    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-tmp-dir\") pod \"premium-w3-88e76d-kserve-6b957679cc-gjgjw\" (UID: \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\") " pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:40:51.883134 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:51.883117    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-home\") pod \"premium-w3-88e76d-kserve-6b957679cc-gjgjw\" (UID: \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\") " pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:40:51.885021 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:51.885001    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-dshm\") pod \"premium-w3-88e76d-kserve-6b957679cc-gjgjw\" (UID: \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\") " pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:40:51.885311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:51.885272    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b10e1d24-9716-4ff6-89c2-e59452b79a8b-tls-certs\") pod \"premium-w3-88e76d-kserve-6b957679cc-gjgjw\" (UID: \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\") " pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:40:51.945513 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:51.945439    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npm5x\" (UniqueName: \"kubernetes.io/projected/b10e1d24-9716-4ff6-89c2-e59452b79a8b-kube-api-access-npm5x\") pod \"premium-w3-88e76d-kserve-6b957679cc-gjgjw\" (UID: \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\") " pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:40:51.964887 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:51.964860    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:40:52.344117 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:52.344092    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"]
Nov 02 01:40:52.346990 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:40:52.346961    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb10e1d24_9716_4ff6_89c2_e59452b79a8b.slice/crio-0e815115c18a55cf20e90c22522c41e9445ae4168fa229debde6c24f8e3383f9 WatchSource:0}: Error finding container 0e815115c18a55cf20e90c22522c41e9445ae4168fa229debde6c24f8e3383f9: Status 404 returned error can't find the container with id 0e815115c18a55cf20e90c22522c41e9445ae4168fa229debde6c24f8e3383f9
Nov 02 01:40:52.890079 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:52.890041    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" event={"ID":"b10e1d24-9716-4ff6-89c2-e59452b79a8b","Type":"ContainerStarted","Data":"23c946c3e4d5cd35fd8834b077c455df9fde1a6a1ae6d260c7446abe64558f17"}
Nov 02 01:40:52.890568 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:52.890089    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" event={"ID":"b10e1d24-9716-4ff6-89c2-e59452b79a8b","Type":"ContainerStarted","Data":"0e815115c18a55cf20e90c22522c41e9445ae4168fa229debde6c24f8e3383f9"}
Nov 02 01:40:52.965219 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:52.965184    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:40:53.007577 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:53.007515    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podStartSLOduration=2.007499041 podStartE2EDuration="2.007499041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:40:53.006755549 +0000 UTC m=+917.910437665" watchObservedRunningTime="2026-11-02 01:40:53.007499041 +0000 UTC m=+917.911181154"
Nov 02 01:40:54.455624 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:54.455585    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4be80e0c-dcsxz"]
Nov 02 01:40:54.459331 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:54.459315    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be80e0c-dcsxz"
Nov 02 01:40:54.513627 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:54.513574    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:40:54.516688 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:54.516659    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4be80e0c-dcsxz"]
Nov 02 01:40:54.563320 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:54.563260    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:40:54.563320 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:54.563308    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:40:54.563320 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:54.563260    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:40:54.615706 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:54.615675    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5605bbd9-39a9-436d-a4c6-d0a73900162e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4be80e0c-dcsxz\" (UID: \"5605bbd9-39a9-436d-a4c6-d0a73900162e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be80e0c-dcsxz"
Nov 02 01:40:54.615885 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:54.615806    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sck9f\" (UniqueName: \"kubernetes.io/projected/5605bbd9-39a9-436d-a4c6-d0a73900162e-kube-api-access-sck9f\") pod \"maas-api-revoke-sub-4be80e0c-dcsxz\" (UID: \"5605bbd9-39a9-436d-a4c6-d0a73900162e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be80e0c-dcsxz"
Nov 02 01:40:54.615885 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:54.615876    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5605bbd9-39a9-436d-a4c6-d0a73900162e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4be80e0c-dcsxz\" (UID: \"5605bbd9-39a9-436d-a4c6-d0a73900162e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be80e0c-dcsxz"
Nov 02 01:40:54.717093 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:54.716998    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5605bbd9-39a9-436d-a4c6-d0a73900162e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4be80e0c-dcsxz\" (UID: \"5605bbd9-39a9-436d-a4c6-d0a73900162e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be80e0c-dcsxz"
Nov 02 01:40:54.717270 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:54.717135    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sck9f\" (UniqueName: \"kubernetes.io/projected/5605bbd9-39a9-436d-a4c6-d0a73900162e-kube-api-access-sck9f\") pod \"maas-api-revoke-sub-4be80e0c-dcsxz\" (UID: \"5605bbd9-39a9-436d-a4c6-d0a73900162e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be80e0c-dcsxz"
Nov 02 01:40:54.717270 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:54.717221    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5605bbd9-39a9-436d-a4c6-d0a73900162e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4be80e0c-dcsxz\" (UID: \"5605bbd9-39a9-436d-a4c6-d0a73900162e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be80e0c-dcsxz"
Nov 02 01:40:54.717728 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:54.717702    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5605bbd9-39a9-436d-a4c6-d0a73900162e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4be80e0c-dcsxz\" (UID: \"5605bbd9-39a9-436d-a4c6-d0a73900162e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be80e0c-dcsxz"
Nov 02 01:40:54.719830 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:54.719805    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5605bbd9-39a9-436d-a4c6-d0a73900162e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4be80e0c-dcsxz\" (UID: \"5605bbd9-39a9-436d-a4c6-d0a73900162e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be80e0c-dcsxz"
Nov 02 01:40:54.778330 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:54.778297    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sck9f\" (UniqueName: \"kubernetes.io/projected/5605bbd9-39a9-436d-a4c6-d0a73900162e-kube-api-access-sck9f\") pod \"maas-api-revoke-sub-4be80e0c-dcsxz\" (UID: \"5605bbd9-39a9-436d-a4c6-d0a73900162e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be80e0c-dcsxz"
Nov 02 01:40:55.073107 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:55.073025    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be80e0c-dcsxz"
Nov 02 01:40:55.446574 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:55.446549    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4be80e0c-dcsxz"]
Nov 02 01:40:55.448624 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:40:55.448595    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5605bbd9_39a9_436d_a4c6_d0a73900162e.slice/crio-29e8ea51c06033887feb94fa9f75e1174380f3055e8653d58a0c48b00955e629 WatchSource:0}: Error finding container 29e8ea51c06033887feb94fa9f75e1174380f3055e8653d58a0c48b00955e629: Status 404 returned error can't find the container with id 29e8ea51c06033887feb94fa9f75e1174380f3055e8653d58a0c48b00955e629
Nov 02 01:40:55.450612 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:55.450596    5639 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:40:55.907499 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:40:55.907462    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be80e0c-dcsxz" event={"ID":"5605bbd9-39a9-436d-a4c6-d0a73900162e","Type":"ContainerStarted","Data":"29e8ea51c06033887feb94fa9f75e1174380f3055e8653d58a0c48b00955e629"}
Nov 02 01:40:57.421341 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:40:57.421305    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:41:01.965464 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:01.965432    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:41:01.978848 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:01.978787    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:41:02.957360 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:02.957326    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:41:06.974175 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:06.974140    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be80e0c-dcsxz" event={"ID":"5605bbd9-39a9-436d-a4c6-d0a73900162e","Type":"ContainerStarted","Data":"dd03b4c27a55809c8f037e3356e32d811b77f4f0d99f8806d81ba3323295981f"}
Nov 02 01:41:07.091029 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:07.090977    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be80e0c-dcsxz" podStartSLOduration=2.375151518 podStartE2EDuration="13.09096307s" totalImagesPullingTime="10.715811552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:40:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:41:07.088779166 +0000 UTC m=+931.992461276" watchObservedRunningTime="2026-11-02 01:41:07.09096307 +0000 UTC m=+931.994645177"
Nov 02 01:41:07.980965 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:07.980932    5639 generic.go:356] "Generic (PLEG): container finished" podID="5605bbd9-39a9-436d-a4c6-d0a73900162e" containerID="dd03b4c27a55809c8f037e3356e32d811b77f4f0d99f8806d81ba3323295981f" exitCode=0
Nov 02 01:41:07.981404 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:07.981013    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be80e0c-dcsxz" event={"ID":"5605bbd9-39a9-436d-a4c6-d0a73900162e","Type":"ContainerDied","Data":"dd03b4c27a55809c8f037e3356e32d811b77f4f0d99f8806d81ba3323295981f"}
Nov 02 01:41:09.137384 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:09.137359    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be80e0c-dcsxz"
Nov 02 01:41:09.197085 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:09.197059    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5605bbd9-39a9-436d-a4c6-d0a73900162e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5605bbd9-39a9-436d-a4c6-d0a73900162e-maas-api-openshift-ca\") pod \"5605bbd9-39a9-436d-a4c6-d0a73900162e\" (UID: \"5605bbd9-39a9-436d-a4c6-d0a73900162e\") "
Nov 02 01:41:09.197222 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:09.197216    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5605bbd9-39a9-436d-a4c6-d0a73900162e-kube-api-access-sck9f\" (UniqueName: \"kubernetes.io/projected/5605bbd9-39a9-436d-a4c6-d0a73900162e-kube-api-access-sck9f\") pod \"5605bbd9-39a9-436d-a4c6-d0a73900162e\" (UID: \"5605bbd9-39a9-436d-a4c6-d0a73900162e\") "
Nov 02 01:41:09.197302 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:09.197254    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5605bbd9-39a9-436d-a4c6-d0a73900162e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5605bbd9-39a9-436d-a4c6-d0a73900162e-maas-api-xks-ca\") pod \"5605bbd9-39a9-436d-a4c6-d0a73900162e\" (UID: \"5605bbd9-39a9-436d-a4c6-d0a73900162e\") "
Nov 02 01:41:09.197397 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:09.197375    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5605bbd9-39a9-436d-a4c6-d0a73900162e-maas-api-openshift-ca" pod "5605bbd9-39a9-436d-a4c6-d0a73900162e" (UID: "5605bbd9-39a9-436d-a4c6-d0a73900162e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:41:09.197591 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:09.197576    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5605bbd9-39a9-436d-a4c6-d0a73900162e-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:41:09.199553 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:09.199528    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5605bbd9-39a9-436d-a4c6-d0a73900162e-kube-api-access-sck9f" pod "5605bbd9-39a9-436d-a4c6-d0a73900162e" (UID: "5605bbd9-39a9-436d-a4c6-d0a73900162e"). InnerVolumeSpecName "kube-api-access-sck9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:41:09.199553 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:09.199545    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5605bbd9-39a9-436d-a4c6-d0a73900162e-maas-api-xks-ca" pod "5605bbd9-39a9-436d-a4c6-d0a73900162e" (UID: "5605bbd9-39a9-436d-a4c6-d0a73900162e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:41:09.298887 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:09.298801    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sck9f\" (UniqueName: \"kubernetes.io/projected/5605bbd9-39a9-436d-a4c6-d0a73900162e-kube-api-access-sck9f\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:41:09.298887 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:09.298828    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5605bbd9-39a9-436d-a4c6-d0a73900162e-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:41:09.992832 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:09.992802    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be80e0c-dcsxz"
Nov 02 01:41:09.993060 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:09.992845    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be80e0c-dcsxz" event={"ID":"5605bbd9-39a9-436d-a4c6-d0a73900162e","Type":"ContainerDied","Data":"29e8ea51c06033887feb94fa9f75e1174380f3055e8653d58a0c48b00955e629"}
Nov 02 01:41:09.993060 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:09.992900    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29e8ea51c06033887feb94fa9f75e1174380f3055e8653d58a0c48b00955e629"
Nov 02 01:41:11.161238 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:11.161207    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:41:27.420543 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:41:27.420509    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:41:35.304513 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:35.304481    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08942016-s2qzs"]
Nov 02 01:41:35.305045 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:35.305031    5639 cpu_manager.go:414] "containerMap: removing container" podUID="5605bbd9-39a9-436d-a4c6-d0a73900162e" containerName="revoke-keys"
Nov 02 01:41:35.305089 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:35.305048    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5605bbd9-39a9-436d-a4c6-d0a73900162e" containerName="revoke-keys"
Nov 02 01:41:35.305166 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:35.305156    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="5605bbd9-39a9-436d-a4c6-d0a73900162e" containerName="revoke-keys"
Nov 02 01:41:35.308513 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:35.308498    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08942016-s2qzs"
Nov 02 01:41:35.338812 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:35.338786    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8657d1b-7690-40be-b28a-a13e6633bb6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08942016-s2qzs\" (UID: \"f8657d1b-7690-40be-b28a-a13e6633bb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08942016-s2qzs"
Nov 02 01:41:35.338812 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:35.338821    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8657d1b-7690-40be-b28a-a13e6633bb6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08942016-s2qzs\" (UID: \"f8657d1b-7690-40be-b28a-a13e6633bb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08942016-s2qzs"
Nov 02 01:41:35.339039 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:35.339018    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgzmr\" (UniqueName: \"kubernetes.io/projected/f8657d1b-7690-40be-b28a-a13e6633bb6b-kube-api-access-hgzmr\") pod \"maas-api-revoke-sub-08942016-s2qzs\" (UID: \"f8657d1b-7690-40be-b28a-a13e6633bb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08942016-s2qzs"
Nov 02 01:41:35.362798 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:35.362764    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:41:35.366142 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:35.366114    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08942016-s2qzs"]
Nov 02 01:41:35.412124 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:35.412091    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:41:35.412263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:35.412148    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:41:35.412442 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:35.412425    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:41:35.440603 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:35.440561    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8657d1b-7690-40be-b28a-a13e6633bb6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08942016-s2qzs\" (UID: \"f8657d1b-7690-40be-b28a-a13e6633bb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08942016-s2qzs"
Nov 02 01:41:35.440603 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:35.440607    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8657d1b-7690-40be-b28a-a13e6633bb6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08942016-s2qzs\" (UID: \"f8657d1b-7690-40be-b28a-a13e6633bb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08942016-s2qzs"
Nov 02 01:41:35.440784 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:35.440694    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgzmr\" (UniqueName: \"kubernetes.io/projected/f8657d1b-7690-40be-b28a-a13e6633bb6b-kube-api-access-hgzmr\") pod \"maas-api-revoke-sub-08942016-s2qzs\" (UID: \"f8657d1b-7690-40be-b28a-a13e6633bb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08942016-s2qzs"
Nov 02 01:41:35.441382 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:35.441359    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8657d1b-7690-40be-b28a-a13e6633bb6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08942016-s2qzs\" (UID: \"f8657d1b-7690-40be-b28a-a13e6633bb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08942016-s2qzs"
Nov 02 01:41:35.443041 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:35.443013    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8657d1b-7690-40be-b28a-a13e6633bb6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08942016-s2qzs\" (UID: \"f8657d1b-7690-40be-b28a-a13e6633bb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08942016-s2qzs"
Nov 02 01:41:35.501463 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:35.501436    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgzmr\" (UniqueName: \"kubernetes.io/projected/f8657d1b-7690-40be-b28a-a13e6633bb6b-kube-api-access-hgzmr\") pod \"maas-api-revoke-sub-08942016-s2qzs\" (UID: \"f8657d1b-7690-40be-b28a-a13e6633bb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08942016-s2qzs"
Nov 02 01:41:35.622190 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:35.622164    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08942016-s2qzs"
Nov 02 01:41:35.999324 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:35.999297    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08942016-s2qzs"]
Nov 02 01:41:36.001260 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:41:36.001233    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8657d1b_7690_40be_b28a_a13e6633bb6b.slice/crio-2065ee6b0f06a1d3c8d7c975c3d26cae2ce0aa3ea67d4211225fc9e76445d434 WatchSource:0}: Error finding container 2065ee6b0f06a1d3c8d7c975c3d26cae2ce0aa3ea67d4211225fc9e76445d434: Status 404 returned error can't find the container with id 2065ee6b0f06a1d3c8d7c975c3d26cae2ce0aa3ea67d4211225fc9e76445d434
Nov 02 01:41:36.146333 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:36.146266    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08942016-s2qzs" event={"ID":"f8657d1b-7690-40be-b28a-a13e6633bb6b","Type":"ContainerStarted","Data":"2065ee6b0f06a1d3c8d7c975c3d26cae2ce0aa3ea67d4211225fc9e76445d434"}
Nov 02 01:41:36.633615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:36.633584    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"]
Nov 02 01:41:36.640520 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:36.640483    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:41:36.694511 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:36.694482    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5888dc\"/\"premium-w6-5888dc-kserve-self-signed-certs\""
Nov 02 01:41:36.708648 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:36.708616    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"]
Nov 02 01:41:36.755092 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:36.755064    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dd080e9-d0c8-4341-ac96-562165899a80-tls-certs\") pod \"premium-w6-5888dc-kserve-69f8b55c64-j7s6h\" (UID: \"0dd080e9-d0c8-4341-ac96-562165899a80\") " pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:41:36.755262 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:36.755114    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c7r7\" (UniqueName: \"kubernetes.io/projected/0dd080e9-d0c8-4341-ac96-562165899a80-kube-api-access-4c7r7\") pod \"premium-w6-5888dc-kserve-69f8b55c64-j7s6h\" (UID: \"0dd080e9-d0c8-4341-ac96-562165899a80\") " pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:41:36.755262 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:36.755162    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-model-cache\") pod \"premium-w6-5888dc-kserve-69f8b55c64-j7s6h\" (UID: \"0dd080e9-d0c8-4341-ac96-562165899a80\") " pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:41:36.755262 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:36.755211    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-home\") pod \"premium-w6-5888dc-kserve-69f8b55c64-j7s6h\" (UID: \"0dd080e9-d0c8-4341-ac96-562165899a80\") " pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:41:36.755409 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:36.755273    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-tmp-dir\") pod \"premium-w6-5888dc-kserve-69f8b55c64-j7s6h\" (UID: \"0dd080e9-d0c8-4341-ac96-562165899a80\") " pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:41:36.755409 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:36.755391    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-dshm\") pod \"premium-w6-5888dc-kserve-69f8b55c64-j7s6h\" (UID: \"0dd080e9-d0c8-4341-ac96-562165899a80\") " pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:41:36.856013 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:36.855983    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-dshm\") pod \"premium-w6-5888dc-kserve-69f8b55c64-j7s6h\" (UID: \"0dd080e9-d0c8-4341-ac96-562165899a80\") " pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:41:36.856188 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:36.856085    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dd080e9-d0c8-4341-ac96-562165899a80-tls-certs\") pod \"premium-w6-5888dc-kserve-69f8b55c64-j7s6h\" (UID: \"0dd080e9-d0c8-4341-ac96-562165899a80\") " pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:41:36.856188 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:36.856129    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c7r7\" (UniqueName: \"kubernetes.io/projected/0dd080e9-d0c8-4341-ac96-562165899a80-kube-api-access-4c7r7\") pod \"premium-w6-5888dc-kserve-69f8b55c64-j7s6h\" (UID: \"0dd080e9-d0c8-4341-ac96-562165899a80\") " pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:41:36.856348 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:36.856326    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-model-cache\") pod \"premium-w6-5888dc-kserve-69f8b55c64-j7s6h\" (UID: \"0dd080e9-d0c8-4341-ac96-562165899a80\") " pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:41:36.856437 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:36.856400    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-home\") pod \"premium-w6-5888dc-kserve-69f8b55c64-j7s6h\" (UID: \"0dd080e9-d0c8-4341-ac96-562165899a80\") " pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:41:36.856489 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:36.856434    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-tmp-dir\") pod \"premium-w6-5888dc-kserve-69f8b55c64-j7s6h\" (UID: \"0dd080e9-d0c8-4341-ac96-562165899a80\") " pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:41:36.856699 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:36.856659    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-model-cache\") pod \"premium-w6-5888dc-kserve-69f8b55c64-j7s6h\" (UID: \"0dd080e9-d0c8-4341-ac96-562165899a80\") " pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:41:36.856699 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:36.856714    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-home\") pod \"premium-w6-5888dc-kserve-69f8b55c64-j7s6h\" (UID: \"0dd080e9-d0c8-4341-ac96-562165899a80\") " pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:41:36.856990 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:36.856853    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-tmp-dir\") pod \"premium-w6-5888dc-kserve-69f8b55c64-j7s6h\" (UID: \"0dd080e9-d0c8-4341-ac96-562165899a80\") " pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:41:36.858671 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:36.858646    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-dshm\") pod \"premium-w6-5888dc-kserve-69f8b55c64-j7s6h\" (UID: \"0dd080e9-d0c8-4341-ac96-562165899a80\") " pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:41:36.859170 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:36.859147    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dd080e9-d0c8-4341-ac96-562165899a80-tls-certs\") pod \"premium-w6-5888dc-kserve-69f8b55c64-j7s6h\" (UID: \"0dd080e9-d0c8-4341-ac96-562165899a80\") " pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:41:36.917714 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:36.917646    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c7r7\" (UniqueName: \"kubernetes.io/projected/0dd080e9-d0c8-4341-ac96-562165899a80-kube-api-access-4c7r7\") pod \"premium-w6-5888dc-kserve-69f8b55c64-j7s6h\" (UID: \"0dd080e9-d0c8-4341-ac96-562165899a80\") " pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:41:36.964731 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:36.964699    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:41:37.153791 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:37.153757    5639 generic.go:356] "Generic (PLEG): container finished" podID="f8657d1b-7690-40be-b28a-a13e6633bb6b" containerID="1afba297197ec82658a3c3b5438e1a606423301874665d95e6d1bed50927d0e6" exitCode=0
Nov 02 01:41:37.153953 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:37.153802    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08942016-s2qzs" event={"ID":"f8657d1b-7690-40be-b28a-a13e6633bb6b","Type":"ContainerDied","Data":"1afba297197ec82658a3c3b5438e1a606423301874665d95e6d1bed50927d0e6"}
Nov 02 01:41:37.383630 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:37.383608    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"]
Nov 02 01:41:37.385677 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:41:37.385644    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0dd080e9_d0c8_4341_ac96_562165899a80.slice/crio-2991ade83cbf6040d3013e0c4dfb234017882f1c1c1a9f8c5768af853faa11ad WatchSource:0}: Error finding container 2991ade83cbf6040d3013e0c4dfb234017882f1c1c1a9f8c5768af853faa11ad: Status 404 returned error can't find the container with id 2991ade83cbf6040d3013e0c4dfb234017882f1c1c1a9f8c5768af853faa11ad
Nov 02 01:41:38.168261 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:38.168216    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" event={"ID":"0dd080e9-d0c8-4341-ac96-562165899a80","Type":"ContainerStarted","Data":"1eae94d5c379dfa2de7082f7beaa829d193398bae412fa0a567a99b565b26efe"}
Nov 02 01:41:38.168261 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:38.168261    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" event={"ID":"0dd080e9-d0c8-4341-ac96-562165899a80","Type":"ContainerStarted","Data":"2991ade83cbf6040d3013e0c4dfb234017882f1c1c1a9f8c5768af853faa11ad"}
Nov 02 01:41:38.306045 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:38.305996    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podStartSLOduration=2.305982807 podStartE2EDuration="2.305982807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:41:38.3051528 +0000 UTC m=+963.208834915" watchObservedRunningTime="2026-11-02 01:41:38.305982807 +0000 UTC m=+963.209665005"
Nov 02 01:41:38.322887 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:38.322862    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08942016-s2qzs"
Nov 02 01:41:38.394205 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:38.394174    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8657d1b-7690-40be-b28a-a13e6633bb6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8657d1b-7690-40be-b28a-a13e6633bb6b-maas-api-xks-ca\") pod \"f8657d1b-7690-40be-b28a-a13e6633bb6b\" (UID: \"f8657d1b-7690-40be-b28a-a13e6633bb6b\") "
Nov 02 01:41:38.394393 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:38.394382    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8657d1b-7690-40be-b28a-a13e6633bb6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8657d1b-7690-40be-b28a-a13e6633bb6b-maas-api-openshift-ca\") pod \"f8657d1b-7690-40be-b28a-a13e6633bb6b\" (UID: \"f8657d1b-7690-40be-b28a-a13e6633bb6b\") "
Nov 02 01:41:38.394438 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:38.394409    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8657d1b-7690-40be-b28a-a13e6633bb6b-kube-api-access-hgzmr\" (UniqueName: \"kubernetes.io/projected/f8657d1b-7690-40be-b28a-a13e6633bb6b-kube-api-access-hgzmr\") pod \"f8657d1b-7690-40be-b28a-a13e6633bb6b\" (UID: \"f8657d1b-7690-40be-b28a-a13e6633bb6b\") "
Nov 02 01:41:38.394726 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:38.394700    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8657d1b-7690-40be-b28a-a13e6633bb6b-maas-api-openshift-ca" pod "f8657d1b-7690-40be-b28a-a13e6633bb6b" (UID: "f8657d1b-7690-40be-b28a-a13e6633bb6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:41:38.396502 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:38.396477    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8657d1b-7690-40be-b28a-a13e6633bb6b-kube-api-access-hgzmr" pod "f8657d1b-7690-40be-b28a-a13e6633bb6b" (UID: "f8657d1b-7690-40be-b28a-a13e6633bb6b"). InnerVolumeSpecName "kube-api-access-hgzmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:41:38.396612 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:38.396593    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8657d1b-7690-40be-b28a-a13e6633bb6b-maas-api-xks-ca" pod "f8657d1b-7690-40be-b28a-a13e6633bb6b" (UID: "f8657d1b-7690-40be-b28a-a13e6633bb6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:41:38.495353 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:38.495260    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8657d1b-7690-40be-b28a-a13e6633bb6b-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:41:38.495353 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:38.495314    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8657d1b-7690-40be-b28a-a13e6633bb6b-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:41:38.495353 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:38.495325    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgzmr\" (UniqueName: \"kubernetes.io/projected/f8657d1b-7690-40be-b28a-a13e6633bb6b-kube-api-access-hgzmr\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:41:38.965439 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:38.965409    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:41:39.175194 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:39.175161    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08942016-s2qzs"
Nov 02 01:41:39.175630 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:39.175197    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08942016-s2qzs" event={"ID":"f8657d1b-7690-40be-b28a-a13e6633bb6b","Type":"ContainerDied","Data":"2065ee6b0f06a1d3c8d7c975c3d26cae2ce0aa3ea67d4211225fc9e76445d434"}
Nov 02 01:41:39.175630 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:39.175244    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2065ee6b0f06a1d3c8d7c975c3d26cae2ce0aa3ea67d4211225fc9e76445d434"
Nov 02 01:41:44.721304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:44.718418    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:41:46.965347 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:46.965305    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:41:46.978309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:46.978246    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:41:47.234467 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:47.234366    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:41:50.902617 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:50.902581    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54fad367-ntrpm"]
Nov 02 01:41:50.903165 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:50.903150    5639 cpu_manager.go:414] "containerMap: removing container" podUID="f8657d1b-7690-40be-b28a-a13e6633bb6b" containerName="revoke-keys"
Nov 02 01:41:50.903204 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:50.903167    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8657d1b-7690-40be-b28a-a13e6633bb6b" containerName="revoke-keys"
Nov 02 01:41:50.903252 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:50.903243    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8657d1b-7690-40be-b28a-a13e6633bb6b" containerName="revoke-keys"
Nov 02 01:41:50.906478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:50.906460    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fad367-ntrpm"
Nov 02 01:41:50.953748 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:50.953713    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpkgj\" (UniqueName: \"kubernetes.io/projected/0d4d547e-c9a3-45ea-a59b-90ecc0627947-kube-api-access-tpkgj\") pod \"maas-api-revoke-sub-54fad367-ntrpm\" (UID: \"0d4d547e-c9a3-45ea-a59b-90ecc0627947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fad367-ntrpm"
Nov 02 01:41:50.953937 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:50.953780    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d4d547e-c9a3-45ea-a59b-90ecc0627947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54fad367-ntrpm\" (UID: \"0d4d547e-c9a3-45ea-a59b-90ecc0627947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fad367-ntrpm"
Nov 02 01:41:50.953989 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:50.953932    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d4d547e-c9a3-45ea-a59b-90ecc0627947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54fad367-ntrpm\" (UID: \"0d4d547e-c9a3-45ea-a59b-90ecc0627947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fad367-ntrpm"
Nov 02 01:41:50.961000 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:50.960960    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:41:50.961192 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:50.960964    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:41:51.010091 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:51.010051    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:41:51.010301 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:51.010167    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:41:51.014863 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:51.014825    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54fad367-ntrpm"]
Nov 02 01:41:51.055103 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:51.055064    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d4d547e-c9a3-45ea-a59b-90ecc0627947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54fad367-ntrpm\" (UID: \"0d4d547e-c9a3-45ea-a59b-90ecc0627947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fad367-ntrpm"
Nov 02 01:41:51.055387 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:51.055142    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpkgj\" (UniqueName: \"kubernetes.io/projected/0d4d547e-c9a3-45ea-a59b-90ecc0627947-kube-api-access-tpkgj\") pod \"maas-api-revoke-sub-54fad367-ntrpm\" (UID: \"0d4d547e-c9a3-45ea-a59b-90ecc0627947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fad367-ntrpm"
Nov 02 01:41:51.055387 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:51.055189    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d4d547e-c9a3-45ea-a59b-90ecc0627947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54fad367-ntrpm\" (UID: \"0d4d547e-c9a3-45ea-a59b-90ecc0627947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fad367-ntrpm"
Nov 02 01:41:51.055778 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:51.055758    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d4d547e-c9a3-45ea-a59b-90ecc0627947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54fad367-ntrpm\" (UID: \"0d4d547e-c9a3-45ea-a59b-90ecc0627947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fad367-ntrpm"
Nov 02 01:41:51.057722 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:51.057702    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d4d547e-c9a3-45ea-a59b-90ecc0627947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54fad367-ntrpm\" (UID: \"0d4d547e-c9a3-45ea-a59b-90ecc0627947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fad367-ntrpm"
Nov 02 01:41:51.122912 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:51.122873    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpkgj\" (UniqueName: \"kubernetes.io/projected/0d4d547e-c9a3-45ea-a59b-90ecc0627947-kube-api-access-tpkgj\") pod \"maas-api-revoke-sub-54fad367-ntrpm\" (UID: \"0d4d547e-c9a3-45ea-a59b-90ecc0627947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fad367-ntrpm"
Nov 02 01:41:51.221037 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:51.220961    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fad367-ntrpm"
Nov 02 01:41:51.632584 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:51.632562    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54fad367-ntrpm"]
Nov 02 01:41:51.635184 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:41:51.635154    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d4d547e_c9a3_45ea_a59b_90ecc0627947.slice/crio-27e10e2752fadcd733e497d4222d348d86828e0f23f4a7604fac64378c4b3431 WatchSource:0}: Error finding container 27e10e2752fadcd733e497d4222d348d86828e0f23f4a7604fac64378c4b3431: Status 404 returned error can't find the container with id 27e10e2752fadcd733e497d4222d348d86828e0f23f4a7604fac64378c4b3431
Nov 02 01:41:52.258434 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:52.258320    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fad367-ntrpm" event={"ID":"0d4d547e-c9a3-45ea-a59b-90ecc0627947","Type":"ContainerStarted","Data":"f80a4dfd341b8781c7b51b2f5d5e686f58b71ffe4bc07b4b2609f25ad7e29ea7"}
Nov 02 01:41:52.258434 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:52.258384    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fad367-ntrpm" event={"ID":"0d4d547e-c9a3-45ea-a59b-90ecc0627947","Type":"ContainerStarted","Data":"27e10e2752fadcd733e497d4222d348d86828e0f23f4a7604fac64378c4b3431"}
Nov 02 01:41:52.374207 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:52.374155    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fad367-ntrpm" podStartSLOduration=2.3741410099999998 podStartE2EDuration="2.37414101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:41:52.373199409 +0000 UTC m=+977.276881523" watchObservedRunningTime="2026-11-02 01:41:52.37414101 +0000 UTC m=+977.277823123"
Nov 02 01:41:53.265979 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:53.265942    5639 generic.go:356] "Generic (PLEG): container finished" podID="0d4d547e-c9a3-45ea-a59b-90ecc0627947" containerID="f80a4dfd341b8781c7b51b2f5d5e686f58b71ffe4bc07b4b2609f25ad7e29ea7" exitCode=0
Nov 02 01:41:53.266405 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:53.266027    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fad367-ntrpm" event={"ID":"0d4d547e-c9a3-45ea-a59b-90ecc0627947","Type":"ContainerDied","Data":"f80a4dfd341b8781c7b51b2f5d5e686f58b71ffe4bc07b4b2609f25ad7e29ea7"}
Nov 02 01:41:54.423642 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:54.423617    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fad367-ntrpm"
Nov 02 01:41:54.497024 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:54.496994    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d4d547e-c9a3-45ea-a59b-90ecc0627947-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d4d547e-c9a3-45ea-a59b-90ecc0627947-maas-api-openshift-ca\") pod \"0d4d547e-c9a3-45ea-a59b-90ecc0627947\" (UID: \"0d4d547e-c9a3-45ea-a59b-90ecc0627947\") "
Nov 02 01:41:54.497189 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:54.497107    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d4d547e-c9a3-45ea-a59b-90ecc0627947-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d4d547e-c9a3-45ea-a59b-90ecc0627947-maas-api-xks-ca\") pod \"0d4d547e-c9a3-45ea-a59b-90ecc0627947\" (UID: \"0d4d547e-c9a3-45ea-a59b-90ecc0627947\") "
Nov 02 01:41:54.497189 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:54.497169    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d4d547e-c9a3-45ea-a59b-90ecc0627947-kube-api-access-tpkgj\" (UniqueName: \"kubernetes.io/projected/0d4d547e-c9a3-45ea-a59b-90ecc0627947-kube-api-access-tpkgj\") pod \"0d4d547e-c9a3-45ea-a59b-90ecc0627947\" (UID: \"0d4d547e-c9a3-45ea-a59b-90ecc0627947\") "
Nov 02 01:41:54.497434 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:54.497402    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d4d547e-c9a3-45ea-a59b-90ecc0627947-maas-api-openshift-ca" pod "0d4d547e-c9a3-45ea-a59b-90ecc0627947" (UID: "0d4d547e-c9a3-45ea-a59b-90ecc0627947"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:41:54.499402 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:54.499378    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d4d547e-c9a3-45ea-a59b-90ecc0627947-maas-api-xks-ca" pod "0d4d547e-c9a3-45ea-a59b-90ecc0627947" (UID: "0d4d547e-c9a3-45ea-a59b-90ecc0627947"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:41:54.499490 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:54.499448    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d4d547e-c9a3-45ea-a59b-90ecc0627947-kube-api-access-tpkgj" pod "0d4d547e-c9a3-45ea-a59b-90ecc0627947" (UID: "0d4d547e-c9a3-45ea-a59b-90ecc0627947"). InnerVolumeSpecName "kube-api-access-tpkgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:41:54.599058 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:54.599020    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpkgj\" (UniqueName: \"kubernetes.io/projected/0d4d547e-c9a3-45ea-a59b-90ecc0627947-kube-api-access-tpkgj\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:41:54.599058 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:54.599059    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d4d547e-c9a3-45ea-a59b-90ecc0627947-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:41:54.599262 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:54.599076    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d4d547e-c9a3-45ea-a59b-90ecc0627947-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:41:55.278615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:55.278588    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fad367-ntrpm"
Nov 02 01:41:55.278788 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:55.278635    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fad367-ntrpm" event={"ID":"0d4d547e-c9a3-45ea-a59b-90ecc0627947","Type":"ContainerDied","Data":"27e10e2752fadcd733e497d4222d348d86828e0f23f4a7604fac64378c4b3431"}
Nov 02 01:41:55.278788 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:41:55.278682    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27e10e2752fadcd733e497d4222d348d86828e0f23f4a7604fac64378c4b3431"
Nov 02 01:41:57.421369 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:41:57.421328    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:42:01.592757 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:01.592722    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04784feb-vctkw"]
Nov 02 01:42:01.593367 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:01.593249    5639 cpu_manager.go:414] "containerMap: removing container" podUID="0d4d547e-c9a3-45ea-a59b-90ecc0627947" containerName="revoke-keys"
Nov 02 01:42:01.593367 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:01.593260    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d4d547e-c9a3-45ea-a59b-90ecc0627947" containerName="revoke-keys"
Nov 02 01:42:01.593367 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:01.593367    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d4d547e-c9a3-45ea-a59b-90ecc0627947" containerName="revoke-keys"
Nov 02 01:42:01.596682 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:01.596662    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04784feb-vctkw"
Nov 02 01:42:01.661015 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:01.660983    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04784feb-vctkw"]
Nov 02 01:42:01.683960 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:01.683920    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56fdfb40-2f99-4420-a0c3-179f8a740250-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04784feb-vctkw\" (UID: \"56fdfb40-2f99-4420-a0c3-179f8a740250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04784feb-vctkw"
Nov 02 01:42:01.684153 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:01.683973    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56fdfb40-2f99-4420-a0c3-179f8a740250-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04784feb-vctkw\" (UID: \"56fdfb40-2f99-4420-a0c3-179f8a740250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04784feb-vctkw"
Nov 02 01:42:01.684226 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:01.684166    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcqxj\" (UniqueName: \"kubernetes.io/projected/56fdfb40-2f99-4420-a0c3-179f8a740250-kube-api-access-xcqxj\") pod \"maas-api-revoke-sub-04784feb-vctkw\" (UID: \"56fdfb40-2f99-4420-a0c3-179f8a740250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04784feb-vctkw"
Nov 02 01:42:01.706492 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:01.706466    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:42:01.706492 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:01.706466    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:42:01.706492 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:01.706466    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:42:01.706492 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:01.706506    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:42:01.785688 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:01.785655    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56fdfb40-2f99-4420-a0c3-179f8a740250-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04784feb-vctkw\" (UID: \"56fdfb40-2f99-4420-a0c3-179f8a740250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04784feb-vctkw"
Nov 02 01:42:01.785901 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:01.785702    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56fdfb40-2f99-4420-a0c3-179f8a740250-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04784feb-vctkw\" (UID: \"56fdfb40-2f99-4420-a0c3-179f8a740250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04784feb-vctkw"
Nov 02 01:42:01.785901 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:01.785798    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcqxj\" (UniqueName: \"kubernetes.io/projected/56fdfb40-2f99-4420-a0c3-179f8a740250-kube-api-access-xcqxj\") pod \"maas-api-revoke-sub-04784feb-vctkw\" (UID: \"56fdfb40-2f99-4420-a0c3-179f8a740250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04784feb-vctkw"
Nov 02 01:42:01.786556 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:01.786525    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56fdfb40-2f99-4420-a0c3-179f8a740250-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04784feb-vctkw\" (UID: \"56fdfb40-2f99-4420-a0c3-179f8a740250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04784feb-vctkw"
Nov 02 01:42:01.788389 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:01.788360    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56fdfb40-2f99-4420-a0c3-179f8a740250-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04784feb-vctkw\" (UID: \"56fdfb40-2f99-4420-a0c3-179f8a740250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04784feb-vctkw"
Nov 02 01:42:01.846601 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:01.846534    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcqxj\" (UniqueName: \"kubernetes.io/projected/56fdfb40-2f99-4420-a0c3-179f8a740250-kube-api-access-xcqxj\") pod \"maas-api-revoke-sub-04784feb-vctkw\" (UID: \"56fdfb40-2f99-4420-a0c3-179f8a740250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04784feb-vctkw"
Nov 02 01:42:01.910949 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:01.910913    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04784feb-vctkw"
Nov 02 01:42:02.294733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:02.294710    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04784feb-vctkw"]
Nov 02 01:42:02.296262 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:42:02.296224    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56fdfb40_2f99_4420_a0c3_179f8a740250.slice/crio-cfa472595efba2132db3fe00d068dea76d6208b899cf99c700f8255783651439 WatchSource:0}: Error finding container cfa472595efba2132db3fe00d068dea76d6208b899cf99c700f8255783651439: Status 404 returned error can't find the container with id cfa472595efba2132db3fe00d068dea76d6208b899cf99c700f8255783651439
Nov 02 01:42:02.322964 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:02.322935    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04784feb-vctkw" event={"ID":"56fdfb40-2f99-4420-a0c3-179f8a740250","Type":"ContainerStarted","Data":"cfa472595efba2132db3fe00d068dea76d6208b899cf99c700f8255783651439"}
Nov 02 01:42:03.330713 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:03.330680    5639 generic.go:356] "Generic (PLEG): container finished" podID="56fdfb40-2f99-4420-a0c3-179f8a740250" containerID="bc93aa7671586f9fbe46585735f2c72c814cf28a6d9d75a77311ab20c299ee07" exitCode=0
Nov 02 01:42:03.331196 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:03.330750    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04784feb-vctkw" event={"ID":"56fdfb40-2f99-4420-a0c3-179f8a740250","Type":"ContainerDied","Data":"bc93aa7671586f9fbe46585735f2c72c814cf28a6d9d75a77311ab20c299ee07"}
Nov 02 01:42:04.487579 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:04.487551    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04784feb-vctkw"
Nov 02 01:42:04.617638 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:04.617549    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56fdfb40-2f99-4420-a0c3-179f8a740250-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56fdfb40-2f99-4420-a0c3-179f8a740250-maas-api-xks-ca\") pod \"56fdfb40-2f99-4420-a0c3-179f8a740250\" (UID: \"56fdfb40-2f99-4420-a0c3-179f8a740250\") "
Nov 02 01:42:04.617796 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:04.617696    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56fdfb40-2f99-4420-a0c3-179f8a740250-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56fdfb40-2f99-4420-a0c3-179f8a740250-maas-api-openshift-ca\") pod \"56fdfb40-2f99-4420-a0c3-179f8a740250\" (UID: \"56fdfb40-2f99-4420-a0c3-179f8a740250\") "
Nov 02 01:42:04.617796 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:04.617733    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56fdfb40-2f99-4420-a0c3-179f8a740250-kube-api-access-xcqxj\" (UniqueName: \"kubernetes.io/projected/56fdfb40-2f99-4420-a0c3-179f8a740250-kube-api-access-xcqxj\") pod \"56fdfb40-2f99-4420-a0c3-179f8a740250\" (UID: \"56fdfb40-2f99-4420-a0c3-179f8a740250\") "
Nov 02 01:42:04.618470 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:04.618101    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56fdfb40-2f99-4420-a0c3-179f8a740250-maas-api-openshift-ca" pod "56fdfb40-2f99-4420-a0c3-179f8a740250" (UID: "56fdfb40-2f99-4420-a0c3-179f8a740250"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:42:04.618470 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:04.618242    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56fdfb40-2f99-4420-a0c3-179f8a740250-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:04.620043 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:04.620020    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56fdfb40-2f99-4420-a0c3-179f8a740250-kube-api-access-xcqxj" pod "56fdfb40-2f99-4420-a0c3-179f8a740250" (UID: "56fdfb40-2f99-4420-a0c3-179f8a740250"). InnerVolumeSpecName "kube-api-access-xcqxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:42:04.620166 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:04.620148    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56fdfb40-2f99-4420-a0c3-179f8a740250-maas-api-xks-ca" pod "56fdfb40-2f99-4420-a0c3-179f8a740250" (UID: "56fdfb40-2f99-4420-a0c3-179f8a740250"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:42:04.719133 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:04.719094    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcqxj\" (UniqueName: \"kubernetes.io/projected/56fdfb40-2f99-4420-a0c3-179f8a740250-kube-api-access-xcqxj\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:04.719133 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:04.719127    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56fdfb40-2f99-4420-a0c3-179f8a740250-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:05.346583 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:05.346549    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04784feb-vctkw"
Nov 02 01:42:05.346583 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:05.346569    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04784feb-vctkw" event={"ID":"56fdfb40-2f99-4420-a0c3-179f8a740250","Type":"ContainerDied","Data":"cfa472595efba2132db3fe00d068dea76d6208b899cf99c700f8255783651439"}
Nov 02 01:42:05.346872 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:05.346626    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfa472595efba2132db3fe00d068dea76d6208b899cf99c700f8255783651439"
Nov 02 01:42:06.845392 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:06.845346    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a8a1d0a-qtms2"]
Nov 02 01:42:06.846078 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:06.846058    5639 cpu_manager.go:414] "containerMap: removing container" podUID="56fdfb40-2f99-4420-a0c3-179f8a740250" containerName="revoke-keys"
Nov 02 01:42:06.846165 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:06.846081    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56fdfb40-2f99-4420-a0c3-179f8a740250" containerName="revoke-keys"
Nov 02 01:42:06.846225 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:06.846169    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="56fdfb40-2f99-4420-a0c3-179f8a740250" containerName="revoke-keys"
Nov 02 01:42:06.850190 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:06.850172    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8a1d0a-qtms2"
Nov 02 01:42:06.947592 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:06.947560    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8dc7940-e89b-4d9c-ab81-2c66fe140456-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a8a1d0a-qtms2\" (UID: \"a8dc7940-e89b-4d9c-ab81-2c66fe140456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8a1d0a-qtms2"
Nov 02 01:42:06.947768 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:06.947662    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8dc7940-e89b-4d9c-ab81-2c66fe140456-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a8a1d0a-qtms2\" (UID: \"a8dc7940-e89b-4d9c-ab81-2c66fe140456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8a1d0a-qtms2"
Nov 02 01:42:06.947843 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:06.947824    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cqcl\" (UniqueName: \"kubernetes.io/projected/a8dc7940-e89b-4d9c-ab81-2c66fe140456-kube-api-access-8cqcl\") pod \"maas-api-revoke-sub-6a8a1d0a-qtms2\" (UID: \"a8dc7940-e89b-4d9c-ab81-2c66fe140456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8a1d0a-qtms2"
Nov 02 01:42:06.952598 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:06.952571    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:42:06.952598 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:06.952571    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:42:06.952797 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:06.952708    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:42:06.952797 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:06.952780    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:42:06.955130 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:06.955108    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a8a1d0a-qtms2"]
Nov 02 01:42:07.049350 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:07.049263    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8dc7940-e89b-4d9c-ab81-2c66fe140456-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a8a1d0a-qtms2\" (UID: \"a8dc7940-e89b-4d9c-ab81-2c66fe140456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8a1d0a-qtms2"
Nov 02 01:42:07.049541 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:07.049372    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8dc7940-e89b-4d9c-ab81-2c66fe140456-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a8a1d0a-qtms2\" (UID: \"a8dc7940-e89b-4d9c-ab81-2c66fe140456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8a1d0a-qtms2"
Nov 02 01:42:07.049541 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:07.049506    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cqcl\" (UniqueName: \"kubernetes.io/projected/a8dc7940-e89b-4d9c-ab81-2c66fe140456-kube-api-access-8cqcl\") pod \"maas-api-revoke-sub-6a8a1d0a-qtms2\" (UID: \"a8dc7940-e89b-4d9c-ab81-2c66fe140456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8a1d0a-qtms2"
Nov 02 01:42:07.050077 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:07.050053    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8dc7940-e89b-4d9c-ab81-2c66fe140456-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a8a1d0a-qtms2\" (UID: \"a8dc7940-e89b-4d9c-ab81-2c66fe140456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8a1d0a-qtms2"
Nov 02 01:42:07.051827 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:07.051798    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8dc7940-e89b-4d9c-ab81-2c66fe140456-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a8a1d0a-qtms2\" (UID: \"a8dc7940-e89b-4d9c-ab81-2c66fe140456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8a1d0a-qtms2"
Nov 02 01:42:07.110776 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:07.110697    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cqcl\" (UniqueName: \"kubernetes.io/projected/a8dc7940-e89b-4d9c-ab81-2c66fe140456-kube-api-access-8cqcl\") pod \"maas-api-revoke-sub-6a8a1d0a-qtms2\" (UID: \"a8dc7940-e89b-4d9c-ab81-2c66fe140456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8a1d0a-qtms2"
Nov 02 01:42:07.165801 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:07.165765    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8a1d0a-qtms2"
Nov 02 01:42:07.545556 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:07.545530    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a8a1d0a-qtms2"]
Nov 02 01:42:07.547765 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:42:07.547738    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8dc7940_e89b_4d9c_ab81_2c66fe140456.slice/crio-8edbcccf738bdfc5a6584fd6dfa774727b31dfff2bf79913c8f960ee685d5472 WatchSource:0}: Error finding container 8edbcccf738bdfc5a6584fd6dfa774727b31dfff2bf79913c8f960ee685d5472: Status 404 returned error can't find the container with id 8edbcccf738bdfc5a6584fd6dfa774727b31dfff2bf79913c8f960ee685d5472
Nov 02 01:42:08.373309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:08.370583    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8a1d0a-qtms2" event={"ID":"a8dc7940-e89b-4d9c-ab81-2c66fe140456","Type":"ContainerStarted","Data":"1a7ed6f01703f7b64b0531a41289c6f2ee045bfb8f58ec0bc8f3ef8a7827b6d8"}
Nov 02 01:42:08.373309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:08.370666    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8a1d0a-qtms2" event={"ID":"a8dc7940-e89b-4d9c-ab81-2c66fe140456","Type":"ContainerStarted","Data":"8edbcccf738bdfc5a6584fd6dfa774727b31dfff2bf79913c8f960ee685d5472"}
Nov 02 01:42:08.502313 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:08.501221    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8a1d0a-qtms2" podStartSLOduration=2.501198408 podStartE2EDuration="2.501198408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:08.494686035 +0000 UTC m=+993.398368161" watchObservedRunningTime="2026-11-02 01:42:08.501198408 +0000 UTC m=+993.404880523"
Nov 02 01:42:09.387317 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:09.384159    5639 generic.go:356] "Generic (PLEG): container finished" podID="a8dc7940-e89b-4d9c-ab81-2c66fe140456" containerID="1a7ed6f01703f7b64b0531a41289c6f2ee045bfb8f58ec0bc8f3ef8a7827b6d8" exitCode=0
Nov 02 01:42:09.387317 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:09.384339    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8a1d0a-qtms2" event={"ID":"a8dc7940-e89b-4d9c-ab81-2c66fe140456","Type":"ContainerDied","Data":"1a7ed6f01703f7b64b0531a41289c6f2ee045bfb8f58ec0bc8f3ef8a7827b6d8"}
Nov 02 01:42:10.670096 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:10.670067    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8a1d0a-qtms2"
Nov 02 01:42:10.752081 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:10.751989    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8dc7940-e89b-4d9c-ab81-2c66fe140456-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8dc7940-e89b-4d9c-ab81-2c66fe140456-maas-api-openshift-ca\") pod \"a8dc7940-e89b-4d9c-ab81-2c66fe140456\" (UID: \"a8dc7940-e89b-4d9c-ab81-2c66fe140456\") "
Nov 02 01:42:10.752228 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:10.752155    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8dc7940-e89b-4d9c-ab81-2c66fe140456-kube-api-access-8cqcl\" (UniqueName: \"kubernetes.io/projected/a8dc7940-e89b-4d9c-ab81-2c66fe140456-kube-api-access-8cqcl\") pod \"a8dc7940-e89b-4d9c-ab81-2c66fe140456\" (UID: \"a8dc7940-e89b-4d9c-ab81-2c66fe140456\") "
Nov 02 01:42:10.752228 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:10.752217    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8dc7940-e89b-4d9c-ab81-2c66fe140456-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8dc7940-e89b-4d9c-ab81-2c66fe140456-maas-api-xks-ca\") pod \"a8dc7940-e89b-4d9c-ab81-2c66fe140456\" (UID: \"a8dc7940-e89b-4d9c-ab81-2c66fe140456\") "
Nov 02 01:42:10.752492 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:10.752463    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8dc7940-e89b-4d9c-ab81-2c66fe140456-maas-api-openshift-ca" pod "a8dc7940-e89b-4d9c-ab81-2c66fe140456" (UID: "a8dc7940-e89b-4d9c-ab81-2c66fe140456"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:42:10.754723 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:10.754681    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8dc7940-e89b-4d9c-ab81-2c66fe140456-maas-api-xks-ca" pod "a8dc7940-e89b-4d9c-ab81-2c66fe140456" (UID: "a8dc7940-e89b-4d9c-ab81-2c66fe140456"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:42:10.754875 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:10.754737    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8dc7940-e89b-4d9c-ab81-2c66fe140456-kube-api-access-8cqcl" pod "a8dc7940-e89b-4d9c-ab81-2c66fe140456" (UID: "a8dc7940-e89b-4d9c-ab81-2c66fe140456"). InnerVolumeSpecName "kube-api-access-8cqcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:42:10.853741 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:10.853702    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8dc7940-e89b-4d9c-ab81-2c66fe140456-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:10.853741 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:10.853738    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8dc7940-e89b-4d9c-ab81-2c66fe140456-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:10.854010 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:10.853755    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cqcl\" (UniqueName: \"kubernetes.io/projected/a8dc7940-e89b-4d9c-ab81-2c66fe140456-kube-api-access-8cqcl\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:11.357501 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:11.357466    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"]
Nov 02 01:42:11.357678 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:11.357659    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt" podUID="182a2c2c-a34a-41c8-bf80-982d15b15d26" containerName="istio-proxy" containerID="cri-o://660e7907ff8148d7166b8f378cee54932601f86b79ae4c3195bf8bcad4c2e3bf" gracePeriod=30
Nov 02 01:42:11.423924 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:11.423893    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8a1d0a-qtms2"
Nov 02 01:42:11.424218 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:11.423914    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8a1d0a-qtms2" event={"ID":"a8dc7940-e89b-4d9c-ab81-2c66fe140456","Type":"ContainerDied","Data":"8edbcccf738bdfc5a6584fd6dfa774727b31dfff2bf79913c8f960ee685d5472"}
Nov 02 01:42:11.424427 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:11.424246    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8edbcccf738bdfc5a6584fd6dfa774727b31dfff2bf79913c8f960ee685d5472"
Nov 02 01:42:12.388255 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:12.388215    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e07cc748-4qgfl"]
Nov 02 01:42:12.389101 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:12.389080    5639 cpu_manager.go:414] "containerMap: removing container" podUID="a8dc7940-e89b-4d9c-ab81-2c66fe140456" containerName="revoke-keys"
Nov 02 01:42:12.389221 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:12.389103    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8dc7940-e89b-4d9c-ab81-2c66fe140456" containerName="revoke-keys"
Nov 02 01:42:12.389221 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:12.389217    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8dc7940-e89b-4d9c-ab81-2c66fe140456" containerName="revoke-keys"
Nov 02 01:42:12.393135 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:12.393110    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07cc748-4qgfl"
Nov 02 01:42:12.445610 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:12.445576    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:42:12.472620 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:12.472581    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e61650-f799-4c4d-8ca6-ef03609fd624-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e07cc748-4qgfl\" (UID: \"03e61650-f799-4c4d-8ca6-ef03609fd624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07cc748-4qgfl"
Nov 02 01:42:12.472819 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:12.472798    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e61650-f799-4c4d-8ca6-ef03609fd624-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e07cc748-4qgfl\" (UID: \"03e61650-f799-4c4d-8ca6-ef03609fd624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07cc748-4qgfl"
Nov 02 01:42:12.472959 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:12.472942    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlbhx\" (UniqueName: \"kubernetes.io/projected/03e61650-f799-4c4d-8ca6-ef03609fd624-kube-api-access-zlbhx\") pod \"maas-api-revoke-sub-e07cc748-4qgfl\" (UID: \"03e61650-f799-4c4d-8ca6-ef03609fd624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07cc748-4qgfl"
Nov 02 01:42:12.495119 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:12.495088    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:42:12.495330 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:12.495300    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:42:12.495863 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:12.495831    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:42:12.501806 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:12.501769    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e07cc748-4qgfl"]
Nov 02 01:42:12.574000 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:12.573968    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e61650-f799-4c4d-8ca6-ef03609fd624-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e07cc748-4qgfl\" (UID: \"03e61650-f799-4c4d-8ca6-ef03609fd624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07cc748-4qgfl"
Nov 02 01:42:12.574467 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:12.574051    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlbhx\" (UniqueName: \"kubernetes.io/projected/03e61650-f799-4c4d-8ca6-ef03609fd624-kube-api-access-zlbhx\") pod \"maas-api-revoke-sub-e07cc748-4qgfl\" (UID: \"03e61650-f799-4c4d-8ca6-ef03609fd624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07cc748-4qgfl"
Nov 02 01:42:12.574467 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:12.574092    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e61650-f799-4c4d-8ca6-ef03609fd624-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e07cc748-4qgfl\" (UID: \"03e61650-f799-4c4d-8ca6-ef03609fd624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07cc748-4qgfl"
Nov 02 01:42:12.574881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:12.574858    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e61650-f799-4c4d-8ca6-ef03609fd624-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e07cc748-4qgfl\" (UID: \"03e61650-f799-4c4d-8ca6-ef03609fd624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07cc748-4qgfl"
Nov 02 01:42:12.576695 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:12.576670    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e61650-f799-4c4d-8ca6-ef03609fd624-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e07cc748-4qgfl\" (UID: \"03e61650-f799-4c4d-8ca6-ef03609fd624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07cc748-4qgfl"
Nov 02 01:42:12.637369 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:12.637318    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlbhx\" (UniqueName: \"kubernetes.io/projected/03e61650-f799-4c4d-8ca6-ef03609fd624-kube-api-access-zlbhx\") pod \"maas-api-revoke-sub-e07cc748-4qgfl\" (UID: \"03e61650-f799-4c4d-8ca6-ef03609fd624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07cc748-4qgfl"
Nov 02 01:42:12.707846 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:12.707761    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07cc748-4qgfl"
Nov 02 01:42:13.096642 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:13.096613    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e07cc748-4qgfl"]
Nov 02 01:42:13.099216 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:42:13.099171    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e61650_f799_4c4d_8ca6_ef03609fd624.slice/crio-7ed6ebebb457a1e2e45a5eb9e246d0f89a6a03a7a5b864eba0a7ae687c461434 WatchSource:0}: Error finding container 7ed6ebebb457a1e2e45a5eb9e246d0f89a6a03a7a5b864eba0a7ae687c461434: Status 404 returned error can't find the container with id 7ed6ebebb457a1e2e45a5eb9e246d0f89a6a03a7a5b864eba0a7ae687c461434
Nov 02 01:42:13.439596 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:13.439556    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07cc748-4qgfl" event={"ID":"03e61650-f799-4c4d-8ca6-ef03609fd624","Type":"ContainerStarted","Data":"a4cbbc43e75ef28929bde42f2983e50b1ab0f1c6e6aa6545a0abc665293e85e1"}
Nov 02 01:42:13.440005 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:13.439623    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07cc748-4qgfl" event={"ID":"03e61650-f799-4c4d-8ca6-ef03609fd624","Type":"ContainerStarted","Data":"7ed6ebebb457a1e2e45a5eb9e246d0f89a6a03a7a5b864eba0a7ae687c461434"}
Nov 02 01:42:13.555017 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:13.554951    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07cc748-4qgfl" podStartSLOduration=1.554931763 podStartE2EDuration="1.554931763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:13.554552858 +0000 UTC m=+998.458234992" watchObservedRunningTime="2026-11-02 01:42:13.554931763 +0000 UTC m=+998.458613878"
Nov 02 01:42:14.447459 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:14.447423    5639 generic.go:356] "Generic (PLEG): container finished" podID="03e61650-f799-4c4d-8ca6-ef03609fd624" containerID="a4cbbc43e75ef28929bde42f2983e50b1ab0f1c6e6aa6545a0abc665293e85e1" exitCode=0
Nov 02 01:42:14.447883 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:14.447485    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07cc748-4qgfl" event={"ID":"03e61650-f799-4c4d-8ca6-ef03609fd624","Type":"ContainerDied","Data":"a4cbbc43e75ef28929bde42f2983e50b1ab0f1c6e6aa6545a0abc665293e85e1"}
Nov 02 01:42:15.614936 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:15.614896    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07cc748-4qgfl"
Nov 02 01:42:15.712637 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:15.712544    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03e61650-f799-4c4d-8ca6-ef03609fd624-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e61650-f799-4c4d-8ca6-ef03609fd624-maas-api-xks-ca\") pod \"03e61650-f799-4c4d-8ca6-ef03609fd624\" (UID: \"03e61650-f799-4c4d-8ca6-ef03609fd624\") "
Nov 02 01:42:15.712637 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:15.712608    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e61650-f799-4c4d-8ca6-ef03609fd624-kube-api-access-zlbhx\" (UniqueName: \"kubernetes.io/projected/03e61650-f799-4c4d-8ca6-ef03609fd624-kube-api-access-zlbhx\") pod \"03e61650-f799-4c4d-8ca6-ef03609fd624\" (UID: \"03e61650-f799-4c4d-8ca6-ef03609fd624\") "
Nov 02 01:42:15.712836 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:15.712721    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03e61650-f799-4c4d-8ca6-ef03609fd624-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e61650-f799-4c4d-8ca6-ef03609fd624-maas-api-openshift-ca\") pod \"03e61650-f799-4c4d-8ca6-ef03609fd624\" (UID: \"03e61650-f799-4c4d-8ca6-ef03609fd624\") "
Nov 02 01:42:15.713124 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:15.713090    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e61650-f799-4c4d-8ca6-ef03609fd624-maas-api-openshift-ca" pod "03e61650-f799-4c4d-8ca6-ef03609fd624" (UID: "03e61650-f799-4c4d-8ca6-ef03609fd624"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:42:15.714788 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:15.714768    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03e61650-f799-4c4d-8ca6-ef03609fd624-maas-api-xks-ca" pod "03e61650-f799-4c4d-8ca6-ef03609fd624" (UID: "03e61650-f799-4c4d-8ca6-ef03609fd624"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:42:15.714881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:15.714866    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e61650-f799-4c4d-8ca6-ef03609fd624-kube-api-access-zlbhx" pod "03e61650-f799-4c4d-8ca6-ef03609fd624" (UID: "03e61650-f799-4c4d-8ca6-ef03609fd624"). InnerVolumeSpecName "kube-api-access-zlbhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:42:15.814985 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:15.814948    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e61650-f799-4c4d-8ca6-ef03609fd624-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:15.814985 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:15.814978    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlbhx\" (UniqueName: \"kubernetes.io/projected/03e61650-f799-4c4d-8ca6-ef03609fd624-kube-api-access-zlbhx\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:15.814985 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:15.814991    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e61650-f799-4c4d-8ca6-ef03609fd624-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:16.460978 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.460945    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07cc748-4qgfl"
Nov 02 01:42:16.461150 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.460985    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07cc748-4qgfl" event={"ID":"03e61650-f799-4c4d-8ca6-ef03609fd624","Type":"ContainerDied","Data":"7ed6ebebb457a1e2e45a5eb9e246d0f89a6a03a7a5b864eba0a7ae687c461434"}
Nov 02 01:42:16.461150 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.461046    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ed6ebebb457a1e2e45a5eb9e246d0f89a6a03a7a5b864eba0a7ae687c461434"
Nov 02 01:42:16.620942 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.620874    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:42:16.722930 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.722896    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-credential-socket\") pod \"182a2c2c-a34a-41c8-bf80-982d15b15d26\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") "
Nov 02 01:42:16.723096 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.722946    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-token\" (UniqueName: \"kubernetes.io/projected/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-token\") pod \"182a2c2c-a34a-41c8-bf80-982d15b15d26\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") "
Nov 02 01:42:16.723096 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.722978    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-envoy\") pod \"182a2c2c-a34a-41c8-bf80-982d15b15d26\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") "
Nov 02 01:42:16.723238 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.723201    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/182a2c2c-a34a-41c8-bf80-982d15b15d26-kube-api-access-f8zdn\" (UniqueName: \"kubernetes.io/projected/182a2c2c-a34a-41c8-bf80-982d15b15d26-kube-api-access-f8zdn\") pod \"182a2c2c-a34a-41c8-bf80-982d15b15d26\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") "
Nov 02 01:42:16.723335 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.723240    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-credential-socket" pod "182a2c2c-a34a-41c8-bf80-982d15b15d26" (UID: "182a2c2c-a34a-41c8-bf80-982d15b15d26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:42:16.723335 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.723246    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-podinfo\") pod \"182a2c2c-a34a-41c8-bf80-982d15b15d26\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") "
Nov 02 01:42:16.723453 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.723352    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-data\") pod \"182a2c2c-a34a-41c8-bf80-982d15b15d26\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") "
Nov 02 01:42:16.723453 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.723411    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-workload-socket\") pod \"182a2c2c-a34a-41c8-bf80-982d15b15d26\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") "
Nov 02 01:42:16.723550 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.723480    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/182a2c2c-a34a-41c8-bf80-982d15b15d26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182a2c2c-a34a-41c8-bf80-982d15b15d26-istiod-ca-cert\") pod \"182a2c2c-a34a-41c8-bf80-982d15b15d26\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") "
Nov 02 01:42:16.723550 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.723517    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-workload-certs\") pod \"182a2c2c-a34a-41c8-bf80-982d15b15d26\" (UID: \"182a2c2c-a34a-41c8-bf80-982d15b15d26\") "
Nov 02 01:42:16.723665 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.723644    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-workload-socket" pod "182a2c2c-a34a-41c8-bf80-982d15b15d26" (UID: "182a2c2c-a34a-41c8-bf80-982d15b15d26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:42:16.723726 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.723638    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-data" pod "182a2c2c-a34a-41c8-bf80-982d15b15d26" (UID: "182a2c2c-a34a-41c8-bf80-982d15b15d26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:42:16.723947 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.723900    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-workload-certs" pod "182a2c2c-a34a-41c8-bf80-982d15b15d26" (UID: "182a2c2c-a34a-41c8-bf80-982d15b15d26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:42:16.724003 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.723960    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/182a2c2c-a34a-41c8-bf80-982d15b15d26-istiod-ca-cert" pod "182a2c2c-a34a-41c8-bf80-982d15b15d26" (UID: "182a2c2c-a34a-41c8-bf80-982d15b15d26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:42:16.724102 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.724086    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182a2c2c-a34a-41c8-bf80-982d15b15d26-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:16.724158 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.724103    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:16.724158 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.724113    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:16.724158 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.724121    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:16.724158 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.724129    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:16.725470 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.725442    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-podinfo" pod "182a2c2c-a34a-41c8-bf80-982d15b15d26" (UID: "182a2c2c-a34a-41c8-bf80-982d15b15d26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:42:16.725597 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.725549    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-envoy" pod "182a2c2c-a34a-41c8-bf80-982d15b15d26" (UID: "182a2c2c-a34a-41c8-bf80-982d15b15d26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:42:16.725896 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.725872    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-token" pod "182a2c2c-a34a-41c8-bf80-982d15b15d26" (UID: "182a2c2c-a34a-41c8-bf80-982d15b15d26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:42:16.725992 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.725910    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182a2c2c-a34a-41c8-bf80-982d15b15d26-kube-api-access-f8zdn" pod "182a2c2c-a34a-41c8-bf80-982d15b15d26" (UID: "182a2c2c-a34a-41c8-bf80-982d15b15d26"). InnerVolumeSpecName "kube-api-access-f8zdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:42:16.825037 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.824994    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:16.825037 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.825032    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:16.825241 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.825049    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182a2c2c-a34a-41c8-bf80-982d15b15d26-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:16.825241 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:16.825092    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8zdn\" (UniqueName: \"kubernetes.io/projected/182a2c2c-a34a-41c8-bf80-982d15b15d26-kube-api-access-f8zdn\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:17.469350 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:17.469314    5639 generic.go:356] "Generic (PLEG): container finished" podID="182a2c2c-a34a-41c8-bf80-982d15b15d26" containerID="660e7907ff8148d7166b8f378cee54932601f86b79ae4c3195bf8bcad4c2e3bf" exitCode=0
Nov 02 01:42:17.469519 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:17.469440    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"
Nov 02 01:42:17.469519 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:17.469445    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt" event={"ID":"182a2c2c-a34a-41c8-bf80-982d15b15d26","Type":"ContainerDied","Data":"660e7907ff8148d7166b8f378cee54932601f86b79ae4c3195bf8bcad4c2e3bf"}
Nov 02 01:42:17.469519 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:17.469515    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt" event={"ID":"182a2c2c-a34a-41c8-bf80-982d15b15d26","Type":"ContainerDied","Data":"c1c02f78bd7004160ba33e16d4f8fa249c3e4143c78aace1bd2a6f8ed80f8ad4"}
Nov 02 01:42:17.469712 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:17.469546    5639 scope.go:122] "RemoveContainer" containerID="660e7907ff8148d7166b8f378cee54932601f86b79ae4c3195bf8bcad4c2e3bf"
Nov 02 01:42:17.479705 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:17.479540    5639 scope.go:122] "RemoveContainer" containerID="660e7907ff8148d7166b8f378cee54932601f86b79ae4c3195bf8bcad4c2e3bf"
Nov 02 01:42:17.479839 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:42:17.479817    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"660e7907ff8148d7166b8f378cee54932601f86b79ae4c3195bf8bcad4c2e3bf\": container with ID starting with 660e7907ff8148d7166b8f378cee54932601f86b79ae4c3195bf8bcad4c2e3bf not found: ID does not exist" containerID="660e7907ff8148d7166b8f378cee54932601f86b79ae4c3195bf8bcad4c2e3bf"
Nov 02 01:42:17.479881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:17.479854    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"660e7907ff8148d7166b8f378cee54932601f86b79ae4c3195bf8bcad4c2e3bf"} err="failed to get container status \"660e7907ff8148d7166b8f378cee54932601f86b79ae4c3195bf8bcad4c2e3bf\": rpc error: code = NotFound desc = could not find container \"660e7907ff8148d7166b8f378cee54932601f86b79ae4c3195bf8bcad4c2e3bf\": container with ID starting with 660e7907ff8148d7166b8f378cee54932601f86b79ae4c3195bf8bcad4c2e3bf not found: ID does not exist"
Nov 02 01:42:17.635643 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:17.635607    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"]
Nov 02 01:42:17.685532 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:17.685503    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t7rwt"]
Nov 02 01:42:19.219701 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:19.219673    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="182a2c2c-a34a-41c8-bf80-982d15b15d26" path="/var/lib/kubelet/pods/182a2c2c-a34a-41c8-bf80-982d15b15d26/volumes"
Nov 02 01:42:27.425789 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:42:27.425748    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:42:31.966472 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:31.966437    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73fc4abc-hpqtf"]
Nov 02 01:42:31.967152 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:31.967131    5639 cpu_manager.go:414] "containerMap: removing container" podUID="03e61650-f799-4c4d-8ca6-ef03609fd624" containerName="revoke-keys"
Nov 02 01:42:31.967269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:31.967155    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e61650-f799-4c4d-8ca6-ef03609fd624" containerName="revoke-keys"
Nov 02 01:42:31.967269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:31.967236    5639 cpu_manager.go:414] "containerMap: removing container" podUID="182a2c2c-a34a-41c8-bf80-982d15b15d26" containerName="istio-proxy"
Nov 02 01:42:31.967269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:31.967245    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="182a2c2c-a34a-41c8-bf80-982d15b15d26" containerName="istio-proxy"
Nov 02 01:42:31.967467 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:31.967396    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="182a2c2c-a34a-41c8-bf80-982d15b15d26" containerName="istio-proxy"
Nov 02 01:42:31.967467 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:31.967413    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e61650-f799-4c4d-8ca6-ef03609fd624" containerName="revoke-keys"
Nov 02 01:42:31.970885 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:31.970864    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fc4abc-hpqtf"
Nov 02 01:42:32.024731 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:32.024690    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:42:32.073975 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:32.073911    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:42:32.074134 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:32.074063    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:42:32.074134 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:32.074105    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:42:32.076932 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:32.076911    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73fc4abc-hpqtf"]
Nov 02 01:42:32.106714 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:32.106677    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rdgr\" (UniqueName: \"kubernetes.io/projected/9c255371-0987-41f7-81b2-28904d6a7b47-kube-api-access-6rdgr\") pod \"maas-api-revoke-sub-73fc4abc-hpqtf\" (UID: \"9c255371-0987-41f7-81b2-28904d6a7b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fc4abc-hpqtf"
Nov 02 01:42:32.106855 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:32.106763    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c255371-0987-41f7-81b2-28904d6a7b47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73fc4abc-hpqtf\" (UID: \"9c255371-0987-41f7-81b2-28904d6a7b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fc4abc-hpqtf"
Nov 02 01:42:32.106896 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:32.106858    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c255371-0987-41f7-81b2-28904d6a7b47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73fc4abc-hpqtf\" (UID: \"9c255371-0987-41f7-81b2-28904d6a7b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fc4abc-hpqtf"
Nov 02 01:42:32.208084 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:32.208047    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rdgr\" (UniqueName: \"kubernetes.io/projected/9c255371-0987-41f7-81b2-28904d6a7b47-kube-api-access-6rdgr\") pod \"maas-api-revoke-sub-73fc4abc-hpqtf\" (UID: \"9c255371-0987-41f7-81b2-28904d6a7b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fc4abc-hpqtf"
Nov 02 01:42:32.208260 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:32.208112    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c255371-0987-41f7-81b2-28904d6a7b47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73fc4abc-hpqtf\" (UID: \"9c255371-0987-41f7-81b2-28904d6a7b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fc4abc-hpqtf"
Nov 02 01:42:32.208260 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:32.208162    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c255371-0987-41f7-81b2-28904d6a7b47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73fc4abc-hpqtf\" (UID: \"9c255371-0987-41f7-81b2-28904d6a7b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fc4abc-hpqtf"
Nov 02 01:42:32.208842 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:32.208818    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c255371-0987-41f7-81b2-28904d6a7b47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73fc4abc-hpqtf\" (UID: \"9c255371-0987-41f7-81b2-28904d6a7b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fc4abc-hpqtf"
Nov 02 01:42:32.210797 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:32.210769    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c255371-0987-41f7-81b2-28904d6a7b47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73fc4abc-hpqtf\" (UID: \"9c255371-0987-41f7-81b2-28904d6a7b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fc4abc-hpqtf"
Nov 02 01:42:32.269384 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:32.269303    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rdgr\" (UniqueName: \"kubernetes.io/projected/9c255371-0987-41f7-81b2-28904d6a7b47-kube-api-access-6rdgr\") pod \"maas-api-revoke-sub-73fc4abc-hpqtf\" (UID: \"9c255371-0987-41f7-81b2-28904d6a7b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fc4abc-hpqtf"
Nov 02 01:42:32.284176 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:32.284149    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fc4abc-hpqtf"
Nov 02 01:42:32.675332 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:32.675301    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73fc4abc-hpqtf"]
Nov 02 01:42:32.677417 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:42:32.677390    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c255371_0987_41f7_81b2_28904d6a7b47.slice/crio-fa948e0c2e58e8ff11116d88c8e123f46d4fd8d55e638af3f36955fd67c2b1d4 WatchSource:0}: Error finding container fa948e0c2e58e8ff11116d88c8e123f46d4fd8d55e638af3f36955fd67c2b1d4: Status 404 returned error can't find the container with id fa948e0c2e58e8ff11116d88c8e123f46d4fd8d55e638af3f36955fd67c2b1d4
Nov 02 01:42:33.576163 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:33.576127    5639 generic.go:356] "Generic (PLEG): container finished" podID="9c255371-0987-41f7-81b2-28904d6a7b47" containerID="1a18c92b2d5e09cfb06ce5c250e98c5287fdc0de5c75088fa3224be92305bedc" exitCode=0
Nov 02 01:42:33.576594 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:33.576176    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fc4abc-hpqtf" event={"ID":"9c255371-0987-41f7-81b2-28904d6a7b47","Type":"ContainerDied","Data":"1a18c92b2d5e09cfb06ce5c250e98c5287fdc0de5c75088fa3224be92305bedc"}
Nov 02 01:42:33.576594 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:33.576221    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fc4abc-hpqtf" event={"ID":"9c255371-0987-41f7-81b2-28904d6a7b47","Type":"ContainerStarted","Data":"fa948e0c2e58e8ff11116d88c8e123f46d4fd8d55e638af3f36955fd67c2b1d4"}
Nov 02 01:42:34.734109 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:34.734081    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fc4abc-hpqtf"
Nov 02 01:42:34.837166 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:34.837079    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c255371-0987-41f7-81b2-28904d6a7b47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c255371-0987-41f7-81b2-28904d6a7b47-maas-api-openshift-ca\") pod \"9c255371-0987-41f7-81b2-28904d6a7b47\" (UID: \"9c255371-0987-41f7-81b2-28904d6a7b47\") "
Nov 02 01:42:34.837366 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:34.837175    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c255371-0987-41f7-81b2-28904d6a7b47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c255371-0987-41f7-81b2-28904d6a7b47-maas-api-xks-ca\") pod \"9c255371-0987-41f7-81b2-28904d6a7b47\" (UID: \"9c255371-0987-41f7-81b2-28904d6a7b47\") "
Nov 02 01:42:34.837366 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:34.837323    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c255371-0987-41f7-81b2-28904d6a7b47-kube-api-access-6rdgr\" (UniqueName: \"kubernetes.io/projected/9c255371-0987-41f7-81b2-28904d6a7b47-kube-api-access-6rdgr\") pod \"9c255371-0987-41f7-81b2-28904d6a7b47\" (UID: \"9c255371-0987-41f7-81b2-28904d6a7b47\") "
Nov 02 01:42:34.837492 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:34.837462    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c255371-0987-41f7-81b2-28904d6a7b47-maas-api-openshift-ca" pod "9c255371-0987-41f7-81b2-28904d6a7b47" (UID: "9c255371-0987-41f7-81b2-28904d6a7b47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:42:34.837720 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:34.837696    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c255371-0987-41f7-81b2-28904d6a7b47-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:34.839487 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:34.839464    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c255371-0987-41f7-81b2-28904d6a7b47-maas-api-xks-ca" pod "9c255371-0987-41f7-81b2-28904d6a7b47" (UID: "9c255371-0987-41f7-81b2-28904d6a7b47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:42:34.839587 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:34.839552    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c255371-0987-41f7-81b2-28904d6a7b47-kube-api-access-6rdgr" pod "9c255371-0987-41f7-81b2-28904d6a7b47" (UID: "9c255371-0987-41f7-81b2-28904d6a7b47"). InnerVolumeSpecName "kube-api-access-6rdgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:42:34.938721 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:34.938681    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c255371-0987-41f7-81b2-28904d6a7b47-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:34.938721 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:34.938716    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rdgr\" (UniqueName: \"kubernetes.io/projected/9c255371-0987-41f7-81b2-28904d6a7b47-kube-api-access-6rdgr\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:35.480606 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:35.480503    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:42:35.590291 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:35.590241    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fc4abc-hpqtf"
Nov 02 01:42:35.590291 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:35.590267    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fc4abc-hpqtf" event={"ID":"9c255371-0987-41f7-81b2-28904d6a7b47","Type":"ContainerDied","Data":"fa948e0c2e58e8ff11116d88c8e123f46d4fd8d55e638af3f36955fd67c2b1d4"}
Nov 02 01:42:35.590509 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:35.590342    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa948e0c2e58e8ff11116d88c8e123f46d4fd8d55e638af3f36955fd67c2b1d4"
Nov 02 01:42:38.257635 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:38.257599    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7411f8a7-j26c2"]
Nov 02 01:42:38.258200 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:38.258182    5639 cpu_manager.go:414] "containerMap: removing container" podUID="9c255371-0987-41f7-81b2-28904d6a7b47" containerName="revoke-keys"
Nov 02 01:42:38.258200 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:38.258200    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c255371-0987-41f7-81b2-28904d6a7b47" containerName="revoke-keys"
Nov 02 01:42:38.258368 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:38.258347    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c255371-0987-41f7-81b2-28904d6a7b47" containerName="revoke-keys"
Nov 02 01:42:38.263574 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:38.263553    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7411f8a7-j26c2"
Nov 02 01:42:38.319014 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:38.318984    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7411f8a7-j26c2"]
Nov 02 01:42:38.364758 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:38.364701    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:42:38.364947 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:38.364883    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:42:38.365011 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:38.364965    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:42:38.365070 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:38.365059    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:42:38.383740 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:38.383708    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2e5596f-58c4-4645-98c7-5d19ce22eb23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7411f8a7-j26c2\" (UID: \"d2e5596f-58c4-4645-98c7-5d19ce22eb23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7411f8a7-j26c2"
Nov 02 01:42:38.383909 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:38.383821    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wr9w\" (UniqueName: \"kubernetes.io/projected/d2e5596f-58c4-4645-98c7-5d19ce22eb23-kube-api-access-8wr9w\") pod \"maas-api-revoke-sub-7411f8a7-j26c2\" (UID: \"d2e5596f-58c4-4645-98c7-5d19ce22eb23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7411f8a7-j26c2"
Nov 02 01:42:38.384014 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:38.383997    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2e5596f-58c4-4645-98c7-5d19ce22eb23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7411f8a7-j26c2\" (UID: \"d2e5596f-58c4-4645-98c7-5d19ce22eb23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7411f8a7-j26c2"
Nov 02 01:42:38.485077 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:38.485038    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wr9w\" (UniqueName: \"kubernetes.io/projected/d2e5596f-58c4-4645-98c7-5d19ce22eb23-kube-api-access-8wr9w\") pod \"maas-api-revoke-sub-7411f8a7-j26c2\" (UID: \"d2e5596f-58c4-4645-98c7-5d19ce22eb23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7411f8a7-j26c2"
Nov 02 01:42:38.485313 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:38.485129    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2e5596f-58c4-4645-98c7-5d19ce22eb23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7411f8a7-j26c2\" (UID: \"d2e5596f-58c4-4645-98c7-5d19ce22eb23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7411f8a7-j26c2"
Nov 02 01:42:38.485313 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:38.485208    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2e5596f-58c4-4645-98c7-5d19ce22eb23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7411f8a7-j26c2\" (UID: \"d2e5596f-58c4-4645-98c7-5d19ce22eb23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7411f8a7-j26c2"
Nov 02 01:42:38.485870 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:38.485844    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2e5596f-58c4-4645-98c7-5d19ce22eb23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7411f8a7-j26c2\" (UID: \"d2e5596f-58c4-4645-98c7-5d19ce22eb23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7411f8a7-j26c2"
Nov 02 01:42:38.487711 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:38.487689    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2e5596f-58c4-4645-98c7-5d19ce22eb23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7411f8a7-j26c2\" (UID: \"d2e5596f-58c4-4645-98c7-5d19ce22eb23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7411f8a7-j26c2"
Nov 02 01:42:38.548037 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:38.547960    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wr9w\" (UniqueName: \"kubernetes.io/projected/d2e5596f-58c4-4645-98c7-5d19ce22eb23-kube-api-access-8wr9w\") pod \"maas-api-revoke-sub-7411f8a7-j26c2\" (UID: \"d2e5596f-58c4-4645-98c7-5d19ce22eb23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7411f8a7-j26c2"
Nov 02 01:42:38.577682 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:38.577635    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7411f8a7-j26c2"
Nov 02 01:42:38.967445 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:42:38.967409    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e5596f_58c4_4645_98c7_5d19ce22eb23.slice/crio-82cc281975123872077955c4c77b0585fdebbd779149848660dc63087cb5252a WatchSource:0}: Error finding container 82cc281975123872077955c4c77b0585fdebbd779149848660dc63087cb5252a: Status 404 returned error can't find the container with id 82cc281975123872077955c4c77b0585fdebbd779149848660dc63087cb5252a
Nov 02 01:42:38.969713 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:38.969683    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7411f8a7-j26c2"]
Nov 02 01:42:39.619362 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:39.619245    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7411f8a7-j26c2" event={"ID":"d2e5596f-58c4-4645-98c7-5d19ce22eb23","Type":"ContainerStarted","Data":"b9e79d4813b5d86da4971039b53ec183ce880b4e55a1a5138d0403c1ae5cff86"}
Nov 02 01:42:39.619362 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:39.619340    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7411f8a7-j26c2" event={"ID":"d2e5596f-58c4-4645-98c7-5d19ce22eb23","Type":"ContainerStarted","Data":"82cc281975123872077955c4c77b0585fdebbd779149848660dc63087cb5252a"}
Nov 02 01:42:39.736021 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:39.735941    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7411f8a7-j26c2" podStartSLOduration=1.735923236 podStartE2EDuration="1.735923236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:39.733908938 +0000 UTC m=+1024.637591051" watchObservedRunningTime="2026-11-02 01:42:39.735923236 +0000 UTC m=+1024.639605351"
Nov 02 01:42:40.627851 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:40.627819    5639 generic.go:356] "Generic (PLEG): container finished" podID="d2e5596f-58c4-4645-98c7-5d19ce22eb23" containerID="b9e79d4813b5d86da4971039b53ec183ce880b4e55a1a5138d0403c1ae5cff86" exitCode=0
Nov 02 01:42:40.628307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:40.627901    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7411f8a7-j26c2" event={"ID":"d2e5596f-58c4-4645-98c7-5d19ce22eb23","Type":"ContainerDied","Data":"b9e79d4813b5d86da4971039b53ec183ce880b4e55a1a5138d0403c1ae5cff86"}
Nov 02 01:42:41.782967 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:41.782941    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7411f8a7-j26c2"
Nov 02 01:42:41.822848 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:41.822817    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2e5596f-58c4-4645-98c7-5d19ce22eb23-kube-api-access-8wr9w\" (UniqueName: \"kubernetes.io/projected/d2e5596f-58c4-4645-98c7-5d19ce22eb23-kube-api-access-8wr9w\") pod \"d2e5596f-58c4-4645-98c7-5d19ce22eb23\" (UID: \"d2e5596f-58c4-4645-98c7-5d19ce22eb23\") "
Nov 02 01:42:41.823020 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:41.822911    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2e5596f-58c4-4645-98c7-5d19ce22eb23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2e5596f-58c4-4645-98c7-5d19ce22eb23-maas-api-xks-ca\") pod \"d2e5596f-58c4-4645-98c7-5d19ce22eb23\" (UID: \"d2e5596f-58c4-4645-98c7-5d19ce22eb23\") "
Nov 02 01:42:41.823069 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:41.823043    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2e5596f-58c4-4645-98c7-5d19ce22eb23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2e5596f-58c4-4645-98c7-5d19ce22eb23-maas-api-openshift-ca\") pod \"d2e5596f-58c4-4645-98c7-5d19ce22eb23\" (UID: \"d2e5596f-58c4-4645-98c7-5d19ce22eb23\") "
Nov 02 01:42:41.823377 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:41.823350    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2e5596f-58c4-4645-98c7-5d19ce22eb23-maas-api-openshift-ca" pod "d2e5596f-58c4-4645-98c7-5d19ce22eb23" (UID: "d2e5596f-58c4-4645-98c7-5d19ce22eb23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:42:41.825235 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:41.825202    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2e5596f-58c4-4645-98c7-5d19ce22eb23-maas-api-xks-ca" pod "d2e5596f-58c4-4645-98c7-5d19ce22eb23" (UID: "d2e5596f-58c4-4645-98c7-5d19ce22eb23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:42:41.825407 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:41.825384    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e5596f-58c4-4645-98c7-5d19ce22eb23-kube-api-access-8wr9w" pod "d2e5596f-58c4-4645-98c7-5d19ce22eb23" (UID: "d2e5596f-58c4-4645-98c7-5d19ce22eb23"). InnerVolumeSpecName "kube-api-access-8wr9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:42:41.924071 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:41.923991    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2e5596f-58c4-4645-98c7-5d19ce22eb23-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:41.924071 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:41.924020    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2e5596f-58c4-4645-98c7-5d19ce22eb23-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:41.924071 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:41.924032    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wr9w\" (UniqueName: \"kubernetes.io/projected/d2e5596f-58c4-4645-98c7-5d19ce22eb23-kube-api-access-8wr9w\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:42.641873 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:42.641838    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7411f8a7-j26c2"
Nov 02 01:42:42.642061 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:42.641867    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7411f8a7-j26c2" event={"ID":"d2e5596f-58c4-4645-98c7-5d19ce22eb23","Type":"ContainerDied","Data":"82cc281975123872077955c4c77b0585fdebbd779149848660dc63087cb5252a"}
Nov 02 01:42:42.642061 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:42.641924    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82cc281975123872077955c4c77b0585fdebbd779149848660dc63087cb5252a"
Nov 02 01:42:47.072848 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:47.072809    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c688bca2-b264c"]
Nov 02 01:42:47.073595 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:47.073571    5639 cpu_manager.go:414] "containerMap: removing container" podUID="d2e5596f-58c4-4645-98c7-5d19ce22eb23" containerName="revoke-keys"
Nov 02 01:42:47.073595 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:47.073596    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2e5596f-58c4-4645-98c7-5d19ce22eb23" containerName="revoke-keys"
Nov 02 01:42:47.073790 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:47.073777    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2e5596f-58c4-4645-98c7-5d19ce22eb23" containerName="revoke-keys"
Nov 02 01:42:47.077202 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:47.077179    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c688bca2-b264c"
Nov 02 01:42:47.180150 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:47.180123    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:42:47.180150 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:47.180126    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:42:47.180422 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:47.180156    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:42:47.180422 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:47.180167    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:42:47.183350 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:47.183320    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c688bca2-b264c"]
Nov 02 01:42:47.187733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:47.187706    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqjxw\" (UniqueName: \"kubernetes.io/projected/679a115f-0425-4323-babf-8c2697236035-kube-api-access-pqjxw\") pod \"maas-api-revoke-sub-c688bca2-b264c\" (UID: \"679a115f-0425-4323-babf-8c2697236035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c688bca2-b264c"
Nov 02 01:42:47.187835 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:47.187782    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679a115f-0425-4323-babf-8c2697236035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c688bca2-b264c\" (UID: \"679a115f-0425-4323-babf-8c2697236035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c688bca2-b264c"
Nov 02 01:42:47.187907 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:47.187893    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679a115f-0425-4323-babf-8c2697236035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c688bca2-b264c\" (UID: \"679a115f-0425-4323-babf-8c2697236035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c688bca2-b264c"
Nov 02 01:42:47.288922 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:47.288890    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqjxw\" (UniqueName: \"kubernetes.io/projected/679a115f-0425-4323-babf-8c2697236035-kube-api-access-pqjxw\") pod \"maas-api-revoke-sub-c688bca2-b264c\" (UID: \"679a115f-0425-4323-babf-8c2697236035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c688bca2-b264c"
Nov 02 01:42:47.289081 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:47.288957    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679a115f-0425-4323-babf-8c2697236035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c688bca2-b264c\" (UID: \"679a115f-0425-4323-babf-8c2697236035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c688bca2-b264c"
Nov 02 01:42:47.289081 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:47.289020    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679a115f-0425-4323-babf-8c2697236035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c688bca2-b264c\" (UID: \"679a115f-0425-4323-babf-8c2697236035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c688bca2-b264c"
Nov 02 01:42:47.289638 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:47.289617    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679a115f-0425-4323-babf-8c2697236035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c688bca2-b264c\" (UID: \"679a115f-0425-4323-babf-8c2697236035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c688bca2-b264c"
Nov 02 01:42:47.291649 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:47.291619    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679a115f-0425-4323-babf-8c2697236035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c688bca2-b264c\" (UID: \"679a115f-0425-4323-babf-8c2697236035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c688bca2-b264c"
Nov 02 01:42:47.349869 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:47.349830    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqjxw\" (UniqueName: \"kubernetes.io/projected/679a115f-0425-4323-babf-8c2697236035-kube-api-access-pqjxw\") pod \"maas-api-revoke-sub-c688bca2-b264c\" (UID: \"679a115f-0425-4323-babf-8c2697236035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c688bca2-b264c"
Nov 02 01:42:47.389950 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:47.389908    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c688bca2-b264c"
Nov 02 01:42:47.769557 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:47.769528    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c688bca2-b264c"]
Nov 02 01:42:47.771200 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:42:47.771175    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679a115f_0425_4323_babf_8c2697236035.slice/crio-58eaac7d2e4d0085db8b3f72e14758d02c0ccdc69ec5770979933f14502196da WatchSource:0}: Error finding container 58eaac7d2e4d0085db8b3f72e14758d02c0ccdc69ec5770979933f14502196da: Status 404 returned error can't find the container with id 58eaac7d2e4d0085db8b3f72e14758d02c0ccdc69ec5770979933f14502196da
Nov 02 01:42:48.596653 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:48.596614    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d24b62b-qxzwb"]
Nov 02 01:42:48.600906 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:48.600881    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d24b62b-qxzwb"
Nov 02 01:42:48.677868 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:48.676929    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d24b62b-qxzwb"]
Nov 02 01:42:48.699819 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:48.699772    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c688bca2-b264c" event={"ID":"679a115f-0425-4323-babf-8c2697236035","Type":"ContainerStarted","Data":"89cfd726dfcdb32b8e7c79ebbec5ee77cfc09951935425962bb67b80cbb8aff1"}
Nov 02 01:42:48.700098 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:48.700070    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c688bca2-b264c" event={"ID":"679a115f-0425-4323-babf-8c2697236035","Type":"ContainerStarted","Data":"58eaac7d2e4d0085db8b3f72e14758d02c0ccdc69ec5770979933f14502196da"}
Nov 02 01:42:48.706615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:48.706577    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d24b62b-qxzwb\" (UID: \"933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d24b62b-qxzwb"
Nov 02 01:42:48.706775 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:48.706704    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9grnl\" (UniqueName: \"kubernetes.io/projected/933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4-kube-api-access-9grnl\") pod \"maas-api-revoke-sub-0d24b62b-qxzwb\" (UID: \"933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d24b62b-qxzwb"
Nov 02 01:42:48.706849 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:48.706781    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d24b62b-qxzwb\" (UID: \"933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d24b62b-qxzwb"
Nov 02 01:42:48.808501 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:48.808467    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d24b62b-qxzwb\" (UID: \"933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d24b62b-qxzwb"
Nov 02 01:42:48.808704 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:48.808547    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9grnl\" (UniqueName: \"kubernetes.io/projected/933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4-kube-api-access-9grnl\") pod \"maas-api-revoke-sub-0d24b62b-qxzwb\" (UID: \"933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d24b62b-qxzwb"
Nov 02 01:42:48.808704 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:48.808616    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d24b62b-qxzwb\" (UID: \"933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d24b62b-qxzwb"
Nov 02 01:42:48.809233 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:48.809209    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d24b62b-qxzwb\" (UID: \"933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d24b62b-qxzwb"
Nov 02 01:42:48.811209 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:48.811186    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d24b62b-qxzwb\" (UID: \"933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d24b62b-qxzwb"
Nov 02 01:42:48.829343 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:48.829255    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c688bca2-b264c" podStartSLOduration=1.829241552 podStartE2EDuration="1.829241552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:48.828657935 +0000 UTC m=+1033.732340050" watchObservedRunningTime="2026-11-02 01:42:48.829241552 +0000 UTC m=+1033.732923666"
Nov 02 01:42:48.869522 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:48.869444    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9grnl\" (UniqueName: \"kubernetes.io/projected/933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4-kube-api-access-9grnl\") pod \"maas-api-revoke-sub-0d24b62b-qxzwb\" (UID: \"933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d24b62b-qxzwb"
Nov 02 01:42:48.915127 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:48.915093    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d24b62b-qxzwb"
Nov 02 01:42:49.294549 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:49.294525    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d24b62b-qxzwb"]
Nov 02 01:42:49.296368 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:42:49.296340    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933a125f_f96c_4f9f_a7d6_7a6e9bdeaee4.slice/crio-74a03e28ef93bdc43d07cffd8f9bdec8ba98f0c1affc3a35426cd111f529f838 WatchSource:0}: Error finding container 74a03e28ef93bdc43d07cffd8f9bdec8ba98f0c1affc3a35426cd111f529f838: Status 404 returned error can't find the container with id 74a03e28ef93bdc43d07cffd8f9bdec8ba98f0c1affc3a35426cd111f529f838
Nov 02 01:42:49.707566 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:49.707528    5639 generic.go:356] "Generic (PLEG): container finished" podID="679a115f-0425-4323-babf-8c2697236035" containerID="89cfd726dfcdb32b8e7c79ebbec5ee77cfc09951935425962bb67b80cbb8aff1" exitCode=0
Nov 02 01:42:49.707998 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:49.707626    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c688bca2-b264c" event={"ID":"679a115f-0425-4323-babf-8c2697236035","Type":"ContainerDied","Data":"89cfd726dfcdb32b8e7c79ebbec5ee77cfc09951935425962bb67b80cbb8aff1"}
Nov 02 01:42:49.709204 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:49.709184    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d24b62b-qxzwb" event={"ID":"933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4","Type":"ContainerStarted","Data":"1e5eced2de1d1e8522d28a6bec2a74dc4f658300df956fe45054776b8eeb3f00"}
Nov 02 01:42:49.709307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:49.709225    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d24b62b-qxzwb" event={"ID":"933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4","Type":"ContainerStarted","Data":"74a03e28ef93bdc43d07cffd8f9bdec8ba98f0c1affc3a35426cd111f529f838"}
Nov 02 01:42:49.939295 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:49.939231    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d24b62b-qxzwb" podStartSLOduration=1.9392144390000001 podStartE2EDuration="1.939214439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:49.937341083 +0000 UTC m=+1034.841023198" watchObservedRunningTime="2026-11-02 01:42:49.939214439 +0000 UTC m=+1034.842896564"
Nov 02 01:42:50.716874 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:50.716835    5639 generic.go:356] "Generic (PLEG): container finished" podID="933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4" containerID="1e5eced2de1d1e8522d28a6bec2a74dc4f658300df956fe45054776b8eeb3f00" exitCode=0
Nov 02 01:42:50.717375 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:50.716923    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d24b62b-qxzwb" event={"ID":"933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4","Type":"ContainerDied","Data":"1e5eced2de1d1e8522d28a6bec2a74dc4f658300df956fe45054776b8eeb3f00"}
Nov 02 01:42:50.892972 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:50.892945    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c688bca2-b264c"
Nov 02 01:42:50.933271 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:50.933239    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/679a115f-0425-4323-babf-8c2697236035-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679a115f-0425-4323-babf-8c2697236035-maas-api-xks-ca\") pod \"679a115f-0425-4323-babf-8c2697236035\" (UID: \"679a115f-0425-4323-babf-8c2697236035\") "
Nov 02 01:42:50.933461 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:50.933349    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/679a115f-0425-4323-babf-8c2697236035-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679a115f-0425-4323-babf-8c2697236035-maas-api-openshift-ca\") pod \"679a115f-0425-4323-babf-8c2697236035\" (UID: \"679a115f-0425-4323-babf-8c2697236035\") "
Nov 02 01:42:50.933461 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:50.933386    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/679a115f-0425-4323-babf-8c2697236035-kube-api-access-pqjxw\" (UniqueName: \"kubernetes.io/projected/679a115f-0425-4323-babf-8c2697236035-kube-api-access-pqjxw\") pod \"679a115f-0425-4323-babf-8c2697236035\" (UID: \"679a115f-0425-4323-babf-8c2697236035\") "
Nov 02 01:42:50.933696 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:50.933669    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679a115f-0425-4323-babf-8c2697236035-maas-api-openshift-ca" pod "679a115f-0425-4323-babf-8c2697236035" (UID: "679a115f-0425-4323-babf-8c2697236035"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:42:50.933828 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:50.933807    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679a115f-0425-4323-babf-8c2697236035-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:50.935678 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:50.935650    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679a115f-0425-4323-babf-8c2697236035-kube-api-access-pqjxw" pod "679a115f-0425-4323-babf-8c2697236035" (UID: "679a115f-0425-4323-babf-8c2697236035"). InnerVolumeSpecName "kube-api-access-pqjxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:42:50.935678 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:50.935671    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/679a115f-0425-4323-babf-8c2697236035-maas-api-xks-ca" pod "679a115f-0425-4323-babf-8c2697236035" (UID: "679a115f-0425-4323-babf-8c2697236035"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:42:51.034863 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:51.034774    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqjxw\" (UniqueName: \"kubernetes.io/projected/679a115f-0425-4323-babf-8c2697236035-kube-api-access-pqjxw\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:51.034863 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:51.034803    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679a115f-0425-4323-babf-8c2697236035-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:51.723949 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:51.723917    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c688bca2-b264c"
Nov 02 01:42:51.724470 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:51.723956    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c688bca2-b264c" event={"ID":"679a115f-0425-4323-babf-8c2697236035","Type":"ContainerDied","Data":"58eaac7d2e4d0085db8b3f72e14758d02c0ccdc69ec5770979933f14502196da"}
Nov 02 01:42:51.724470 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:51.724004    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58eaac7d2e4d0085db8b3f72e14758d02c0ccdc69ec5770979933f14502196da"
Nov 02 01:42:51.875686 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:51.875661    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d24b62b-qxzwb"
Nov 02 01:42:51.943121 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:51.943091    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4-maas-api-xks-ca\") pod \"933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4\" (UID: \"933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4\") "
Nov 02 01:42:51.943370 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:51.943161    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4-kube-api-access-9grnl\" (UniqueName: \"kubernetes.io/projected/933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4-kube-api-access-9grnl\") pod \"933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4\" (UID: \"933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4\") "
Nov 02 01:42:51.943370 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:51.943312    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4-maas-api-openshift-ca\") pod \"933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4\" (UID: \"933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4\") "
Nov 02 01:42:51.943687 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:51.943657    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4-maas-api-openshift-ca" pod "933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4" (UID: "933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:42:51.945601 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:51.945578    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4-kube-api-access-9grnl" pod "933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4" (UID: "933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4"). InnerVolumeSpecName "kube-api-access-9grnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:42:51.945747 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:51.945731    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4-maas-api-xks-ca" pod "933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4" (UID: "933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:42:52.045208 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:52.045121    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:52.045208 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:52.045154    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9grnl\" (UniqueName: \"kubernetes.io/projected/933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4-kube-api-access-9grnl\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:52.045208 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:52.045165    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:52.731030 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:52.731002    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d24b62b-qxzwb"
Nov 02 01:42:52.731441 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:52.731004    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d24b62b-qxzwb" event={"ID":"933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4","Type":"ContainerDied","Data":"74a03e28ef93bdc43d07cffd8f9bdec8ba98f0c1affc3a35426cd111f529f838"}
Nov 02 01:42:52.731441 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:52.731126    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74a03e28ef93bdc43d07cffd8f9bdec8ba98f0c1affc3a35426cd111f529f838"
Nov 02 01:42:56.586836 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:56.586805    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-667f49cf-k6mxc"]
Nov 02 01:42:56.587383 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:56.587368    5639 cpu_manager.go:414] "containerMap: removing container" podUID="933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4" containerName="revoke-keys"
Nov 02 01:42:56.587433 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:56.587384    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4" containerName="revoke-keys"
Nov 02 01:42:56.587433 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:56.587420    5639 cpu_manager.go:414] "containerMap: removing container" podUID="679a115f-0425-4323-babf-8c2697236035" containerName="revoke-keys"
Nov 02 01:42:56.587433 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:56.587425    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="679a115f-0425-4323-babf-8c2697236035" containerName="revoke-keys"
Nov 02 01:42:56.587534 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:56.587524    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4" containerName="revoke-keys"
Nov 02 01:42:56.587566 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:56.587541    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="679a115f-0425-4323-babf-8c2697236035" containerName="revoke-keys"
Nov 02 01:42:56.590933 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:56.590917    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-667f49cf-k6mxc"
Nov 02 01:42:56.648069 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:56.648029    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-667f49cf-k6mxc"]
Nov 02 01:42:56.693456 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:56.693425    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:42:56.693605 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:56.693461    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:42:56.693605 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:56.693526    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:42:56.696254 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:56.696240    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:42:56.710487 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:56.710448    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c909f3b-9a71-4f93-838c-1e86d9c41897-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-667f49cf-k6mxc\" (UID: \"9c909f3b-9a71-4f93-838c-1e86d9c41897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-667f49cf-k6mxc"
Nov 02 01:42:56.710487 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:56.710485    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfq6p\" (UniqueName: \"kubernetes.io/projected/9c909f3b-9a71-4f93-838c-1e86d9c41897-kube-api-access-jfq6p\") pod \"maas-api-revoke-sub-667f49cf-k6mxc\" (UID: \"9c909f3b-9a71-4f93-838c-1e86d9c41897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-667f49cf-k6mxc"
Nov 02 01:42:56.710950 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:56.710927    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c909f3b-9a71-4f93-838c-1e86d9c41897-maas-api-xks-ca\") pod \"maas-api-revoke-sub-667f49cf-k6mxc\" (UID: \"9c909f3b-9a71-4f93-838c-1e86d9c41897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-667f49cf-k6mxc"
Nov 02 01:42:56.811990 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:56.811954    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c909f3b-9a71-4f93-838c-1e86d9c41897-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-667f49cf-k6mxc\" (UID: \"9c909f3b-9a71-4f93-838c-1e86d9c41897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-667f49cf-k6mxc"
Nov 02 01:42:56.811990 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:56.811991    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfq6p\" (UniqueName: \"kubernetes.io/projected/9c909f3b-9a71-4f93-838c-1e86d9c41897-kube-api-access-jfq6p\") pod \"maas-api-revoke-sub-667f49cf-k6mxc\" (UID: \"9c909f3b-9a71-4f93-838c-1e86d9c41897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-667f49cf-k6mxc"
Nov 02 01:42:56.812229 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:56.812083    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c909f3b-9a71-4f93-838c-1e86d9c41897-maas-api-xks-ca\") pod \"maas-api-revoke-sub-667f49cf-k6mxc\" (UID: \"9c909f3b-9a71-4f93-838c-1e86d9c41897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-667f49cf-k6mxc"
Nov 02 01:42:56.812763 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:56.812737    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c909f3b-9a71-4f93-838c-1e86d9c41897-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-667f49cf-k6mxc\" (UID: \"9c909f3b-9a71-4f93-838c-1e86d9c41897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-667f49cf-k6mxc"
Nov 02 01:42:56.814663 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:56.814641    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c909f3b-9a71-4f93-838c-1e86d9c41897-maas-api-xks-ca\") pod \"maas-api-revoke-sub-667f49cf-k6mxc\" (UID: \"9c909f3b-9a71-4f93-838c-1e86d9c41897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-667f49cf-k6mxc"
Nov 02 01:42:56.873259 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:56.873179    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfq6p\" (UniqueName: \"kubernetes.io/projected/9c909f3b-9a71-4f93-838c-1e86d9c41897-kube-api-access-jfq6p\") pod \"maas-api-revoke-sub-667f49cf-k6mxc\" (UID: \"9c909f3b-9a71-4f93-838c-1e86d9c41897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-667f49cf-k6mxc"
Nov 02 01:42:56.912982 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:56.912947    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-667f49cf-k6mxc"
Nov 02 01:42:57.291435 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:57.291408    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-667f49cf-k6mxc"]
Nov 02 01:42:57.293734 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:42:57.293700    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c909f3b_9a71_4f93_838c_1e86d9c41897.slice/crio-2191ac614d6f9e9a7833618bfa4a59740b9ce68664d0682ce16b4140be488d25 WatchSource:0}: Error finding container 2191ac614d6f9e9a7833618bfa4a59740b9ce68664d0682ce16b4140be488d25: Status 404 returned error can't find the container with id 2191ac614d6f9e9a7833618bfa4a59740b9ce68664d0682ce16b4140be488d25
Nov 02 01:42:57.471916 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:42:57.471872    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:42:57.764677 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:57.764636    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-667f49cf-k6mxc" event={"ID":"9c909f3b-9a71-4f93-838c-1e86d9c41897","Type":"ContainerStarted","Data":"a2f6b4985511549b042efd6849126ae556e3ea33d98e8adea2d601b53d1b63c1"}
Nov 02 01:42:57.765092 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:57.764701    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-667f49cf-k6mxc" event={"ID":"9c909f3b-9a71-4f93-838c-1e86d9c41897","Type":"ContainerStarted","Data":"2191ac614d6f9e9a7833618bfa4a59740b9ce68664d0682ce16b4140be488d25"}
Nov 02 01:42:57.816729 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:57.816699    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9347a0a-7g2r8"]
Nov 02 01:42:57.820745 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:57.820722    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9347a0a-7g2r8"
Nov 02 01:42:57.878166 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:57.878131    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9347a0a-7g2r8"]
Nov 02 01:42:57.897220 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:57.897162    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-667f49cf-k6mxc" podStartSLOduration=1.897147854 podStartE2EDuration="1.897147854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:57.894522111 +0000 UTC m=+1042.798204228" watchObservedRunningTime="2026-11-02 01:42:57.897147854 +0000 UTC m=+1042.800829983"
Nov 02 01:42:57.926087 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:57.926008    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64lrw\" (UniqueName: \"kubernetes.io/projected/67cf0050-d217-4871-aa43-afab78812551-kube-api-access-64lrw\") pod \"maas-api-revoke-sub-e9347a0a-7g2r8\" (UID: \"67cf0050-d217-4871-aa43-afab78812551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9347a0a-7g2r8"
Nov 02 01:42:57.926087 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:57.926076    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67cf0050-d217-4871-aa43-afab78812551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9347a0a-7g2r8\" (UID: \"67cf0050-d217-4871-aa43-afab78812551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9347a0a-7g2r8"
Nov 02 01:42:57.926296 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:57.926208    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67cf0050-d217-4871-aa43-afab78812551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9347a0a-7g2r8\" (UID: \"67cf0050-d217-4871-aa43-afab78812551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9347a0a-7g2r8"
Nov 02 01:42:58.027093 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:58.027055    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64lrw\" (UniqueName: \"kubernetes.io/projected/67cf0050-d217-4871-aa43-afab78812551-kube-api-access-64lrw\") pod \"maas-api-revoke-sub-e9347a0a-7g2r8\" (UID: \"67cf0050-d217-4871-aa43-afab78812551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9347a0a-7g2r8"
Nov 02 01:42:58.027326 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:58.027153    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67cf0050-d217-4871-aa43-afab78812551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9347a0a-7g2r8\" (UID: \"67cf0050-d217-4871-aa43-afab78812551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9347a0a-7g2r8"
Nov 02 01:42:58.027326 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:58.027244    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67cf0050-d217-4871-aa43-afab78812551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9347a0a-7g2r8\" (UID: \"67cf0050-d217-4871-aa43-afab78812551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9347a0a-7g2r8"
Nov 02 01:42:58.027987 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:58.027961    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67cf0050-d217-4871-aa43-afab78812551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9347a0a-7g2r8\" (UID: \"67cf0050-d217-4871-aa43-afab78812551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9347a0a-7g2r8"
Nov 02 01:42:58.029844 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:58.029816    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67cf0050-d217-4871-aa43-afab78812551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9347a0a-7g2r8\" (UID: \"67cf0050-d217-4871-aa43-afab78812551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9347a0a-7g2r8"
Nov 02 01:42:58.088633 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:58.088599    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64lrw\" (UniqueName: \"kubernetes.io/projected/67cf0050-d217-4871-aa43-afab78812551-kube-api-access-64lrw\") pod \"maas-api-revoke-sub-e9347a0a-7g2r8\" (UID: \"67cf0050-d217-4871-aa43-afab78812551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9347a0a-7g2r8"
Nov 02 01:42:58.151751 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:58.151624    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9347a0a-7g2r8"
Nov 02 01:42:58.568516 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:58.568488    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9347a0a-7g2r8"]
Nov 02 01:42:58.570315 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:42:58.570246    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67cf0050_d217_4871_aa43_afab78812551.slice/crio-98f0b0ccde26bd5c29492c15b22e020731828e08b3b4e2a8000102884b394f4c WatchSource:0}: Error finding container 98f0b0ccde26bd5c29492c15b22e020731828e08b3b4e2a8000102884b394f4c: Status 404 returned error can't find the container with id 98f0b0ccde26bd5c29492c15b22e020731828e08b3b4e2a8000102884b394f4c
Nov 02 01:42:58.772195 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:58.772164    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9347a0a-7g2r8" event={"ID":"67cf0050-d217-4871-aa43-afab78812551","Type":"ContainerStarted","Data":"98f0b0ccde26bd5c29492c15b22e020731828e08b3b4e2a8000102884b394f4c"}
Nov 02 01:42:58.773758 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:58.773733    5639 generic.go:356] "Generic (PLEG): container finished" podID="9c909f3b-9a71-4f93-838c-1e86d9c41897" containerID="a2f6b4985511549b042efd6849126ae556e3ea33d98e8adea2d601b53d1b63c1" exitCode=0
Nov 02 01:42:58.773889 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:58.773797    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-667f49cf-k6mxc" event={"ID":"9c909f3b-9a71-4f93-838c-1e86d9c41897","Type":"ContainerDied","Data":"a2f6b4985511549b042efd6849126ae556e3ea33d98e8adea2d601b53d1b63c1"}
Nov 02 01:42:59.511995 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:59.511962    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fc9016b-lx48f"]
Nov 02 01:42:59.516155 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:59.516137    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9016b-lx48f"
Nov 02 01:42:59.578274 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:59.578242    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fc9016b-lx48f"]
Nov 02 01:42:59.649440 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:59.649403    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a385d2ad-22c8-4f1e-b98b-9ead1eb4019d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fc9016b-lx48f\" (UID: \"a385d2ad-22c8-4f1e-b98b-9ead1eb4019d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9016b-lx48f"
Nov 02 01:42:59.649625 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:59.649473    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbr9j\" (UniqueName: \"kubernetes.io/projected/a385d2ad-22c8-4f1e-b98b-9ead1eb4019d-kube-api-access-bbr9j\") pod \"maas-api-revoke-sub-8fc9016b-lx48f\" (UID: \"a385d2ad-22c8-4f1e-b98b-9ead1eb4019d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9016b-lx48f"
Nov 02 01:42:59.649625 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:59.649566    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a385d2ad-22c8-4f1e-b98b-9ead1eb4019d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fc9016b-lx48f\" (UID: \"a385d2ad-22c8-4f1e-b98b-9ead1eb4019d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9016b-lx48f"
Nov 02 01:42:59.750799 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:59.750752    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a385d2ad-22c8-4f1e-b98b-9ead1eb4019d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fc9016b-lx48f\" (UID: \"a385d2ad-22c8-4f1e-b98b-9ead1eb4019d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9016b-lx48f"
Nov 02 01:42:59.750961 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:59.750826    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbr9j\" (UniqueName: \"kubernetes.io/projected/a385d2ad-22c8-4f1e-b98b-9ead1eb4019d-kube-api-access-bbr9j\") pod \"maas-api-revoke-sub-8fc9016b-lx48f\" (UID: \"a385d2ad-22c8-4f1e-b98b-9ead1eb4019d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9016b-lx48f"
Nov 02 01:42:59.750961 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:59.750855    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a385d2ad-22c8-4f1e-b98b-9ead1eb4019d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fc9016b-lx48f\" (UID: \"a385d2ad-22c8-4f1e-b98b-9ead1eb4019d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9016b-lx48f"
Nov 02 01:42:59.751602 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:59.751576    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a385d2ad-22c8-4f1e-b98b-9ead1eb4019d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fc9016b-lx48f\" (UID: \"a385d2ad-22c8-4f1e-b98b-9ead1eb4019d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9016b-lx48f"
Nov 02 01:42:59.753273 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:59.753249    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a385d2ad-22c8-4f1e-b98b-9ead1eb4019d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fc9016b-lx48f\" (UID: \"a385d2ad-22c8-4f1e-b98b-9ead1eb4019d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9016b-lx48f"
Nov 02 01:42:59.784154 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:59.784072    5639 generic.go:356] "Generic (PLEG): container finished" podID="67cf0050-d217-4871-aa43-afab78812551" containerID="bebd2e8d892d44bc60243e2c27409d3171bb3167c302e3549552a65c41166a6d" exitCode=0
Nov 02 01:42:59.784708 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:59.784683    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9347a0a-7g2r8" event={"ID":"67cf0050-d217-4871-aa43-afab78812551","Type":"ContainerDied","Data":"bebd2e8d892d44bc60243e2c27409d3171bb3167c302e3549552a65c41166a6d"}
Nov 02 01:42:59.812300 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:59.812244    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbr9j\" (UniqueName: \"kubernetes.io/projected/a385d2ad-22c8-4f1e-b98b-9ead1eb4019d-kube-api-access-bbr9j\") pod \"maas-api-revoke-sub-8fc9016b-lx48f\" (UID: \"a385d2ad-22c8-4f1e-b98b-9ead1eb4019d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9016b-lx48f"
Nov 02 01:42:59.838417 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:59.838384    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9016b-lx48f"
Nov 02 01:42:59.989356 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:42:59.989328    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-667f49cf-k6mxc"
Nov 02 01:43:00.054805 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:00.054710    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c909f3b-9a71-4f93-838c-1e86d9c41897-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c909f3b-9a71-4f93-838c-1e86d9c41897-maas-api-openshift-ca\") pod \"9c909f3b-9a71-4f93-838c-1e86d9c41897\" (UID: \"9c909f3b-9a71-4f93-838c-1e86d9c41897\") "
Nov 02 01:43:00.054805 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:00.054775    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c909f3b-9a71-4f93-838c-1e86d9c41897-kube-api-access-jfq6p\" (UniqueName: \"kubernetes.io/projected/9c909f3b-9a71-4f93-838c-1e86d9c41897-kube-api-access-jfq6p\") pod \"9c909f3b-9a71-4f93-838c-1e86d9c41897\" (UID: \"9c909f3b-9a71-4f93-838c-1e86d9c41897\") "
Nov 02 01:43:00.055037 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:00.054836    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c909f3b-9a71-4f93-838c-1e86d9c41897-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c909f3b-9a71-4f93-838c-1e86d9c41897-maas-api-xks-ca\") pod \"9c909f3b-9a71-4f93-838c-1e86d9c41897\" (UID: \"9c909f3b-9a71-4f93-838c-1e86d9c41897\") "
Nov 02 01:43:00.055104 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:00.055080    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c909f3b-9a71-4f93-838c-1e86d9c41897-maas-api-openshift-ca" pod "9c909f3b-9a71-4f93-838c-1e86d9c41897" (UID: "9c909f3b-9a71-4f93-838c-1e86d9c41897"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:00.055323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:00.055307    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c909f3b-9a71-4f93-838c-1e86d9c41897-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:00.057068 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:00.057047    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c909f3b-9a71-4f93-838c-1e86d9c41897-maas-api-xks-ca" pod "9c909f3b-9a71-4f93-838c-1e86d9c41897" (UID: "9c909f3b-9a71-4f93-838c-1e86d9c41897"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:00.057213 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:00.057192    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c909f3b-9a71-4f93-838c-1e86d9c41897-kube-api-access-jfq6p" pod "9c909f3b-9a71-4f93-838c-1e86d9c41897" (UID: "9c909f3b-9a71-4f93-838c-1e86d9c41897"). InnerVolumeSpecName "kube-api-access-jfq6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:00.156272 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:00.156235    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfq6p\" (UniqueName: \"kubernetes.io/projected/9c909f3b-9a71-4f93-838c-1e86d9c41897-kube-api-access-jfq6p\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:00.156272 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:00.156269    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c909f3b-9a71-4f93-838c-1e86d9c41897-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:00.255973 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:00.255945    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fc9016b-lx48f"]
Nov 02 01:43:00.257274 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:43:00.257250    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda385d2ad_22c8_4f1e_b98b_9ead1eb4019d.slice/crio-84def47cabf873c887814a54bb4a462f2c15ab0347b8fb297d684a7d254da0b5 WatchSource:0}: Error finding container 84def47cabf873c887814a54bb4a462f2c15ab0347b8fb297d684a7d254da0b5: Status 404 returned error can't find the container with id 84def47cabf873c887814a54bb4a462f2c15ab0347b8fb297d684a7d254da0b5
Nov 02 01:43:00.791975 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:00.791889    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-667f49cf-k6mxc"
Nov 02 01:43:00.791975 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:00.791932    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-667f49cf-k6mxc" event={"ID":"9c909f3b-9a71-4f93-838c-1e86d9c41897","Type":"ContainerDied","Data":"2191ac614d6f9e9a7833618bfa4a59740b9ce68664d0682ce16b4140be488d25"}
Nov 02 01:43:00.792517 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:00.791989    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2191ac614d6f9e9a7833618bfa4a59740b9ce68664d0682ce16b4140be488d25"
Nov 02 01:43:00.793782 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:00.793740    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9016b-lx48f" event={"ID":"a385d2ad-22c8-4f1e-b98b-9ead1eb4019d","Type":"ContainerStarted","Data":"3659dba05dcc908d64e2a188b9db3fa1b57bfcc464f41ae1ba06e64132555280"}
Nov 02 01:43:00.793905 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:00.793802    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9016b-lx48f" event={"ID":"a385d2ad-22c8-4f1e-b98b-9ead1eb4019d","Type":"ContainerStarted","Data":"84def47cabf873c887814a54bb4a462f2c15ab0347b8fb297d684a7d254da0b5"}
Nov 02 01:43:00.912391 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:00.912344    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9016b-lx48f" podStartSLOduration=1.912328997 podStartE2EDuration="1.912328997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:00.90937749 +0000 UTC m=+1045.813059605" watchObservedRunningTime="2026-11-02 01:43:00.912328997 +0000 UTC m=+1045.816011111"
Nov 02 01:43:00.938196 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:00.938171    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9347a0a-7g2r8"
Nov 02 01:43:01.065410 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:01.065316    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67cf0050-d217-4871-aa43-afab78812551-kube-api-access-64lrw\" (UniqueName: \"kubernetes.io/projected/67cf0050-d217-4871-aa43-afab78812551-kube-api-access-64lrw\") pod \"67cf0050-d217-4871-aa43-afab78812551\" (UID: \"67cf0050-d217-4871-aa43-afab78812551\") "
Nov 02 01:43:01.065572 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:01.065467    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67cf0050-d217-4871-aa43-afab78812551-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67cf0050-d217-4871-aa43-afab78812551-maas-api-openshift-ca\") pod \"67cf0050-d217-4871-aa43-afab78812551\" (UID: \"67cf0050-d217-4871-aa43-afab78812551\") "
Nov 02 01:43:01.065649 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:01.065618    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67cf0050-d217-4871-aa43-afab78812551-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67cf0050-d217-4871-aa43-afab78812551-maas-api-xks-ca\") pod \"67cf0050-d217-4871-aa43-afab78812551\" (UID: \"67cf0050-d217-4871-aa43-afab78812551\") "
Nov 02 01:43:01.065852 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:01.065822    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67cf0050-d217-4871-aa43-afab78812551-maas-api-openshift-ca" pod "67cf0050-d217-4871-aa43-afab78812551" (UID: "67cf0050-d217-4871-aa43-afab78812551"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:01.066038 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:01.066018    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67cf0050-d217-4871-aa43-afab78812551-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.067635 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:01.067611    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67cf0050-d217-4871-aa43-afab78812551-kube-api-access-64lrw" pod "67cf0050-d217-4871-aa43-afab78812551" (UID: "67cf0050-d217-4871-aa43-afab78812551"). InnerVolumeSpecName "kube-api-access-64lrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:01.067767 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:01.067750    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67cf0050-d217-4871-aa43-afab78812551-maas-api-xks-ca" pod "67cf0050-d217-4871-aa43-afab78812551" (UID: "67cf0050-d217-4871-aa43-afab78812551"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:01.166927 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:01.166900    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67cf0050-d217-4871-aa43-afab78812551-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.166927 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:01.166930    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64lrw\" (UniqueName: \"kubernetes.io/projected/67cf0050-d217-4871-aa43-afab78812551-kube-api-access-64lrw\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.800528 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:01.800498    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9347a0a-7g2r8"
Nov 02 01:43:01.800940 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:01.800536    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9347a0a-7g2r8" event={"ID":"67cf0050-d217-4871-aa43-afab78812551","Type":"ContainerDied","Data":"98f0b0ccde26bd5c29492c15b22e020731828e08b3b4e2a8000102884b394f4c"}
Nov 02 01:43:01.800940 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:01.800589    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98f0b0ccde26bd5c29492c15b22e020731828e08b3b4e2a8000102884b394f4c"
Nov 02 01:43:01.802202 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:01.802174    5639 generic.go:356] "Generic (PLEG): container finished" podID="a385d2ad-22c8-4f1e-b98b-9ead1eb4019d" containerID="3659dba05dcc908d64e2a188b9db3fa1b57bfcc464f41ae1ba06e64132555280" exitCode=0
Nov 02 01:43:01.802352 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:01.802263    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9016b-lx48f" event={"ID":"a385d2ad-22c8-4f1e-b98b-9ead1eb4019d","Type":"ContainerDied","Data":"3659dba05dcc908d64e2a188b9db3fa1b57bfcc464f41ae1ba06e64132555280"}
Nov 02 01:43:02.712512 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.712459    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"]
Nov 02 01:43:02.713063 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.713048    5639 cpu_manager.go:414] "containerMap: removing container" podUID="9c909f3b-9a71-4f93-838c-1e86d9c41897" containerName="revoke-keys"
Nov 02 01:43:02.713108 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.713065    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c909f3b-9a71-4f93-838c-1e86d9c41897" containerName="revoke-keys"
Nov 02 01:43:02.713108 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.713080    5639 cpu_manager.go:414] "containerMap: removing container" podUID="67cf0050-d217-4871-aa43-afab78812551" containerName="revoke-keys"
Nov 02 01:43:02.713108 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.713086    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67cf0050-d217-4871-aa43-afab78812551" containerName="revoke-keys"
Nov 02 01:43:02.713205 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.713172    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="67cf0050-d217-4871-aa43-afab78812551" containerName="revoke-keys"
Nov 02 01:43:02.713205 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.713188    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c909f3b-9a71-4f93-838c-1e86d9c41897" containerName="revoke-keys"
Nov 02 01:43:02.716607 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.716588    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:43:02.786352 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.786317    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-dshm\") pod \"distinct-w1-f276c3-kserve-7d8f59cfc9-psw77\" (UID: \"4939edba-ee67-41c9-aa16-809fb1fc7116\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:43:02.786520 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.786382    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-home\") pod \"distinct-w1-f276c3-kserve-7d8f59cfc9-psw77\" (UID: \"4939edba-ee67-41c9-aa16-809fb1fc7116\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:43:02.786520 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.786417    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-model-cache\") pod \"distinct-w1-f276c3-kserve-7d8f59cfc9-psw77\" (UID: \"4939edba-ee67-41c9-aa16-809fb1fc7116\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:43:02.786520 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.786436    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4939edba-ee67-41c9-aa16-809fb1fc7116-tls-certs\") pod \"distinct-w1-f276c3-kserve-7d8f59cfc9-psw77\" (UID: \"4939edba-ee67-41c9-aa16-809fb1fc7116\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:43:02.786686 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.786523    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsjfp\" (UniqueName: \"kubernetes.io/projected/4939edba-ee67-41c9-aa16-809fb1fc7116-kube-api-access-fsjfp\") pod \"distinct-w1-f276c3-kserve-7d8f59cfc9-psw77\" (UID: \"4939edba-ee67-41c9-aa16-809fb1fc7116\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:43:02.786744 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.786719    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-tmp-dir\") pod \"distinct-w1-f276c3-kserve-7d8f59cfc9-psw77\" (UID: \"4939edba-ee67-41c9-aa16-809fb1fc7116\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:43:02.820418 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.820390    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f276c3\"/\"kube-root-ca.crt\""
Nov 02 01:43:02.820908 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.820640    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f276c3\"/\"openshift-service-ca.crt\""
Nov 02 01:43:02.820908 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.820810    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f276c3\"/\"default-dockercfg-sphkv\""
Nov 02 01:43:02.820908 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.820841    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f276c3\"/\"distinct-w1-f276c3-kserve-self-signed-certs\""
Nov 02 01:43:02.824145 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.824122    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"]
Nov 02 01:43:02.888567 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.888222    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-dshm\") pod \"distinct-w1-f276c3-kserve-7d8f59cfc9-psw77\" (UID: \"4939edba-ee67-41c9-aa16-809fb1fc7116\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:43:02.888892 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.888848    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-home\") pod \"distinct-w1-f276c3-kserve-7d8f59cfc9-psw77\" (UID: \"4939edba-ee67-41c9-aa16-809fb1fc7116\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:43:02.888994 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.888914    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-model-cache\") pod \"distinct-w1-f276c3-kserve-7d8f59cfc9-psw77\" (UID: \"4939edba-ee67-41c9-aa16-809fb1fc7116\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:43:02.888994 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.888946    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4939edba-ee67-41c9-aa16-809fb1fc7116-tls-certs\") pod \"distinct-w1-f276c3-kserve-7d8f59cfc9-psw77\" (UID: \"4939edba-ee67-41c9-aa16-809fb1fc7116\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:43:02.888994 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.888973    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsjfp\" (UniqueName: \"kubernetes.io/projected/4939edba-ee67-41c9-aa16-809fb1fc7116-kube-api-access-fsjfp\") pod \"distinct-w1-f276c3-kserve-7d8f59cfc9-psw77\" (UID: \"4939edba-ee67-41c9-aa16-809fb1fc7116\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:43:02.889184 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.889156    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-tmp-dir\") pod \"distinct-w1-f276c3-kserve-7d8f59cfc9-psw77\" (UID: \"4939edba-ee67-41c9-aa16-809fb1fc7116\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:43:02.889872 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.889645    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-model-cache\") pod \"distinct-w1-f276c3-kserve-7d8f59cfc9-psw77\" (UID: \"4939edba-ee67-41c9-aa16-809fb1fc7116\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:43:02.889872 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.889209    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-home\") pod \"distinct-w1-f276c3-kserve-7d8f59cfc9-psw77\" (UID: \"4939edba-ee67-41c9-aa16-809fb1fc7116\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:43:02.890410 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.890392    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-tmp-dir\") pod \"distinct-w1-f276c3-kserve-7d8f59cfc9-psw77\" (UID: \"4939edba-ee67-41c9-aa16-809fb1fc7116\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:43:02.893930 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.893850    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4939edba-ee67-41c9-aa16-809fb1fc7116-tls-certs\") pod \"distinct-w1-f276c3-kserve-7d8f59cfc9-psw77\" (UID: \"4939edba-ee67-41c9-aa16-809fb1fc7116\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:43:02.895009 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.894754    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-dshm\") pod \"distinct-w1-f276c3-kserve-7d8f59cfc9-psw77\" (UID: \"4939edba-ee67-41c9-aa16-809fb1fc7116\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:43:02.952609 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.952575    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsjfp\" (UniqueName: \"kubernetes.io/projected/4939edba-ee67-41c9-aa16-809fb1fc7116-kube-api-access-fsjfp\") pod \"distinct-w1-f276c3-kserve-7d8f59cfc9-psw77\" (UID: \"4939edba-ee67-41c9-aa16-809fb1fc7116\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:43:02.983017 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:02.982983    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9016b-lx48f"
Nov 02 01:43:03.031295 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:03.031249    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:43:03.092528 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:03.091830    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a385d2ad-22c8-4f1e-b98b-9ead1eb4019d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a385d2ad-22c8-4f1e-b98b-9ead1eb4019d-maas-api-openshift-ca\") pod \"a385d2ad-22c8-4f1e-b98b-9ead1eb4019d\" (UID: \"a385d2ad-22c8-4f1e-b98b-9ead1eb4019d\") "
Nov 02 01:43:03.092528 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:03.091898    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a385d2ad-22c8-4f1e-b98b-9ead1eb4019d-kube-api-access-bbr9j\" (UniqueName: \"kubernetes.io/projected/a385d2ad-22c8-4f1e-b98b-9ead1eb4019d-kube-api-access-bbr9j\") pod \"a385d2ad-22c8-4f1e-b98b-9ead1eb4019d\" (UID: \"a385d2ad-22c8-4f1e-b98b-9ead1eb4019d\") "
Nov 02 01:43:03.092528 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:03.092062    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a385d2ad-22c8-4f1e-b98b-9ead1eb4019d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a385d2ad-22c8-4f1e-b98b-9ead1eb4019d-maas-api-xks-ca\") pod \"a385d2ad-22c8-4f1e-b98b-9ead1eb4019d\" (UID: \"a385d2ad-22c8-4f1e-b98b-9ead1eb4019d\") "
Nov 02 01:43:03.092528 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:03.092219    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a385d2ad-22c8-4f1e-b98b-9ead1eb4019d-maas-api-openshift-ca" pod "a385d2ad-22c8-4f1e-b98b-9ead1eb4019d" (UID: "a385d2ad-22c8-4f1e-b98b-9ead1eb4019d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:03.092814 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:03.092576    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a385d2ad-22c8-4f1e-b98b-9ead1eb4019d-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:03.096457 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:03.096397    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a385d2ad-22c8-4f1e-b98b-9ead1eb4019d-kube-api-access-bbr9j" pod "a385d2ad-22c8-4f1e-b98b-9ead1eb4019d" (UID: "a385d2ad-22c8-4f1e-b98b-9ead1eb4019d"). InnerVolumeSpecName "kube-api-access-bbr9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:03.096712 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:03.096681    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a385d2ad-22c8-4f1e-b98b-9ead1eb4019d-maas-api-xks-ca" pod "a385d2ad-22c8-4f1e-b98b-9ead1eb4019d" (UID: "a385d2ad-22c8-4f1e-b98b-9ead1eb4019d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:03.194458 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:03.194389    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a385d2ad-22c8-4f1e-b98b-9ead1eb4019d-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:03.194458 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:03.194428    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbr9j\" (UniqueName: \"kubernetes.io/projected/a385d2ad-22c8-4f1e-b98b-9ead1eb4019d-kube-api-access-bbr9j\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:03.414129 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:03.414102    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"]
Nov 02 01:43:03.415736 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:43:03.415708    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4939edba_ee67_41c9_aa16_809fb1fc7116.slice/crio-eba1e026998c97557e95f0fef7dc1b2c1159ce75898eae615797eeb5a77c1556 WatchSource:0}: Error finding container eba1e026998c97557e95f0fef7dc1b2c1159ce75898eae615797eeb5a77c1556: Status 404 returned error can't find the container with id eba1e026998c97557e95f0fef7dc1b2c1159ce75898eae615797eeb5a77c1556
Nov 02 01:43:03.816687 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:03.816644    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" event={"ID":"4939edba-ee67-41c9-aa16-809fb1fc7116","Type":"ContainerStarted","Data":"21d5cf11d1e6a02323a20b2b7460b26be6b711d9bbad6302edc720e41a83c3ad"}
Nov 02 01:43:03.816969 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:03.816696    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" event={"ID":"4939edba-ee67-41c9-aa16-809fb1fc7116","Type":"ContainerStarted","Data":"eba1e026998c97557e95f0fef7dc1b2c1159ce75898eae615797eeb5a77c1556"}
Nov 02 01:43:03.818140 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:03.818118    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9016b-lx48f"
Nov 02 01:43:03.818255 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:03.818118    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9016b-lx48f" event={"ID":"a385d2ad-22c8-4f1e-b98b-9ead1eb4019d","Type":"ContainerDied","Data":"84def47cabf873c887814a54bb4a462f2c15ab0347b8fb297d684a7d254da0b5"}
Nov 02 01:43:03.818255 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:03.818244    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84def47cabf873c887814a54bb4a462f2c15ab0347b8fb297d684a7d254da0b5"
Nov 02 01:43:03.935494 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:03.935439    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podStartSLOduration=1.9354216389999999 podStartE2EDuration="1.935421639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:03.933897508 +0000 UTC m=+1048.837579623" watchObservedRunningTime="2026-11-02 01:43:03.935421639 +0000 UTC m=+1048.839103820"
Nov 02 01:43:04.032312 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:04.032255    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:43:04.705323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:04.705292    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8edfdd95-qjp8v"]
Nov 02 01:43:04.705810 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:04.705798    5639 cpu_manager.go:414] "containerMap: removing container" podUID="a385d2ad-22c8-4f1e-b98b-9ead1eb4019d" containerName="revoke-keys"
Nov 02 01:43:04.705855 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:04.705812    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a385d2ad-22c8-4f1e-b98b-9ead1eb4019d" containerName="revoke-keys"
Nov 02 01:43:04.705936 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:04.705926    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="a385d2ad-22c8-4f1e-b98b-9ead1eb4019d" containerName="revoke-keys"
Nov 02 01:43:04.709086 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:04.709068    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edfdd95-qjp8v"
Nov 02 01:43:04.773968 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:04.773938    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:43:04.774180 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:04.774162    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:43:04.774261 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:04.774233    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:43:04.774519 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:04.774491    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:43:04.775750 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:04.775727    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8edfdd95-qjp8v"]
Nov 02 01:43:04.812002 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:04.811966    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b4600f4-8f54-4755-815d-dd4e041065a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8edfdd95-qjp8v\" (UID: \"6b4600f4-8f54-4755-815d-dd4e041065a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edfdd95-qjp8v"
Nov 02 01:43:04.812173 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:04.812017    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b4600f4-8f54-4755-815d-dd4e041065a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8edfdd95-qjp8v\" (UID: \"6b4600f4-8f54-4755-815d-dd4e041065a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edfdd95-qjp8v"
Nov 02 01:43:04.812231 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:04.812213    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fttql\" (UniqueName: \"kubernetes.io/projected/6b4600f4-8f54-4755-815d-dd4e041065a3-kube-api-access-fttql\") pod \"maas-api-revoke-sub-8edfdd95-qjp8v\" (UID: \"6b4600f4-8f54-4755-815d-dd4e041065a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edfdd95-qjp8v"
Nov 02 01:43:04.913467 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:04.913436    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fttql\" (UniqueName: \"kubernetes.io/projected/6b4600f4-8f54-4755-815d-dd4e041065a3-kube-api-access-fttql\") pod \"maas-api-revoke-sub-8edfdd95-qjp8v\" (UID: \"6b4600f4-8f54-4755-815d-dd4e041065a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edfdd95-qjp8v"
Nov 02 01:43:04.913691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:04.913552    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b4600f4-8f54-4755-815d-dd4e041065a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8edfdd95-qjp8v\" (UID: \"6b4600f4-8f54-4755-815d-dd4e041065a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edfdd95-qjp8v"
Nov 02 01:43:04.913691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:04.913578    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b4600f4-8f54-4755-815d-dd4e041065a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8edfdd95-qjp8v\" (UID: \"6b4600f4-8f54-4755-815d-dd4e041065a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edfdd95-qjp8v"
Nov 02 01:43:04.914178 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:04.914155    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b4600f4-8f54-4755-815d-dd4e041065a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8edfdd95-qjp8v\" (UID: \"6b4600f4-8f54-4755-815d-dd4e041065a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edfdd95-qjp8v"
Nov 02 01:43:04.916138 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:04.916117    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b4600f4-8f54-4755-815d-dd4e041065a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8edfdd95-qjp8v\" (UID: \"6b4600f4-8f54-4755-815d-dd4e041065a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edfdd95-qjp8v"
Nov 02 01:43:04.992558 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:04.992481    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fttql\" (UniqueName: \"kubernetes.io/projected/6b4600f4-8f54-4755-815d-dd4e041065a3-kube-api-access-fttql\") pod \"maas-api-revoke-sub-8edfdd95-qjp8v\" (UID: \"6b4600f4-8f54-4755-815d-dd4e041065a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edfdd95-qjp8v"
Nov 02 01:43:05.022641 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:05.022598    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edfdd95-qjp8v"
Nov 02 01:43:05.397845 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:05.397818    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8edfdd95-qjp8v"]
Nov 02 01:43:05.398634 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:43:05.398608    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b4600f4_8f54_4755_815d_dd4e041065a3.slice/crio-676b905ef4e591c1e55ca15f1127d60d0696af564277629cfd9a87bf23660c3f WatchSource:0}: Error finding container 676b905ef4e591c1e55ca15f1127d60d0696af564277629cfd9a87bf23660c3f: Status 404 returned error can't find the container with id 676b905ef4e591c1e55ca15f1127d60d0696af564277629cfd9a87bf23660c3f
Nov 02 01:43:05.832534 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:05.832501    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edfdd95-qjp8v" event={"ID":"6b4600f4-8f54-4755-815d-dd4e041065a3","Type":"ContainerStarted","Data":"3fc8f30239fe3d46d9377fddfb41b3380d1b00a5b5105254c1820c1c3fd9e2f9"}
Nov 02 01:43:05.832717 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:05.832554    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edfdd95-qjp8v" event={"ID":"6b4600f4-8f54-4755-815d-dd4e041065a3","Type":"ContainerStarted","Data":"676b905ef4e591c1e55ca15f1127d60d0696af564277629cfd9a87bf23660c3f"}
Nov 02 01:43:05.962956 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:05.962853    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edfdd95-qjp8v" podStartSLOduration=1.962837622 podStartE2EDuration="1.962837622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:05.960585828 +0000 UTC m=+1050.864267945" watchObservedRunningTime="2026-11-02 01:43:05.962837622 +0000 UTC m=+1050.866519735"
Nov 02 01:43:06.841718 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:06.841674    5639 generic.go:356] "Generic (PLEG): container finished" podID="6b4600f4-8f54-4755-815d-dd4e041065a3" containerID="3fc8f30239fe3d46d9377fddfb41b3380d1b00a5b5105254c1820c1c3fd9e2f9" exitCode=0
Nov 02 01:43:06.842212 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:06.841740    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edfdd95-qjp8v" event={"ID":"6b4600f4-8f54-4755-815d-dd4e041065a3","Type":"ContainerDied","Data":"3fc8f30239fe3d46d9377fddfb41b3380d1b00a5b5105254c1820c1c3fd9e2f9"}
Nov 02 01:43:07.045540 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.045506    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"]
Nov 02 01:43:07.050477 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.050450    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.107339 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.107232    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-f276c3-openshift-default-dockercfg-72fq4\""
Nov 02 01:43:07.111608 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.111577    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"]
Nov 02 01:43:07.137717 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.137677    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ac256a-961a-4ff2-b454-06a201660154-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.137926 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.137828    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.137926 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.137919    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ac256a-961a-4ff2-b454-06a201660154-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.138026 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.137947    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.138026 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.137990    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb2lq\" (UniqueName: \"kubernetes.io/projected/50ac256a-961a-4ff2-b454-06a201660154-kube-api-access-gb2lq\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.138156 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.138131    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.138201 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.138174    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.138260 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.138248    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.138337 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.138317    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50ac256a-961a-4ff2-b454-06a201660154-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.239263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.239230    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ac256a-961a-4ff2-b454-06a201660154-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.239263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.239269    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.239536 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.239342    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ac256a-961a-4ff2-b454-06a201660154-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.239536 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.239360    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.239536 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.239391    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb2lq\" (UniqueName: \"kubernetes.io/projected/50ac256a-961a-4ff2-b454-06a201660154-kube-api-access-gb2lq\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.239536 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.239507    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.239766 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.239544    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.239766 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.239581    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.239766 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.239608    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50ac256a-961a-4ff2-b454-06a201660154-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.239766 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.239750    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.239996 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.239909    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.240056 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.240033    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.240129 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.240106    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.240786 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.240763    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ac256a-961a-4ff2-b454-06a201660154-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.241923 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.241892    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.242033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.242023    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ac256a-961a-4ff2-b454-06a201660154-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.300594 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.300554    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50ac256a-961a-4ff2-b454-06a201660154-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.300741 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.300662    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb2lq\" (UniqueName: \"kubernetes.io/projected/50ac256a-961a-4ff2-b454-06a201660154-kube-api-access-gb2lq\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.364972 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.364892    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:07.798358 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.798188    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"]
Nov 02 01:43:07.800547 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:43:07.800514    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ac256a_961a_4ff2_b454_06a201660154.slice/crio-1766f6f1efdd556b190c4b3f433b53ff3501e6f66cf2e4147242ae45c82f502b WatchSource:0}: Error finding container 1766f6f1efdd556b190c4b3f433b53ff3501e6f66cf2e4147242ae45c82f502b: Status 404 returned error can't find the container with id 1766f6f1efdd556b190c4b3f433b53ff3501e6f66cf2e4147242ae45c82f502b
Nov 02 01:43:07.803044 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.803012    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:43:07.803134 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.803076    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:43:07.803134 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.803106    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:43:07.849685 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.849658    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5" event={"ID":"50ac256a-961a-4ff2-b454-06a201660154","Type":"ContainerStarted","Data":"1766f6f1efdd556b190c4b3f433b53ff3501e6f66cf2e4147242ae45c82f502b"}
Nov 02 01:43:07.999500 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:07.999474    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edfdd95-qjp8v"
Nov 02 01:43:08.051314 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:08.049369    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b4600f4-8f54-4755-815d-dd4e041065a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b4600f4-8f54-4755-815d-dd4e041065a3-maas-api-openshift-ca\") pod \"6b4600f4-8f54-4755-815d-dd4e041065a3\" (UID: \"6b4600f4-8f54-4755-815d-dd4e041065a3\") "
Nov 02 01:43:08.051314 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:08.049445    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b4600f4-8f54-4755-815d-dd4e041065a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b4600f4-8f54-4755-815d-dd4e041065a3-maas-api-xks-ca\") pod \"6b4600f4-8f54-4755-815d-dd4e041065a3\" (UID: \"6b4600f4-8f54-4755-815d-dd4e041065a3\") "
Nov 02 01:43:08.051314 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:08.049511    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b4600f4-8f54-4755-815d-dd4e041065a3-kube-api-access-fttql\" (UniqueName: \"kubernetes.io/projected/6b4600f4-8f54-4755-815d-dd4e041065a3-kube-api-access-fttql\") pod \"6b4600f4-8f54-4755-815d-dd4e041065a3\" (UID: \"6b4600f4-8f54-4755-815d-dd4e041065a3\") "
Nov 02 01:43:08.051314 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:08.051034    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b4600f4-8f54-4755-815d-dd4e041065a3-maas-api-openshift-ca" pod "6b4600f4-8f54-4755-815d-dd4e041065a3" (UID: "6b4600f4-8f54-4755-815d-dd4e041065a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:08.061530 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:08.061485    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b4600f4-8f54-4755-815d-dd4e041065a3-maas-api-xks-ca" pod "6b4600f4-8f54-4755-815d-dd4e041065a3" (UID: "6b4600f4-8f54-4755-815d-dd4e041065a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:08.061777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:08.061735    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b4600f4-8f54-4755-815d-dd4e041065a3-kube-api-access-fttql" pod "6b4600f4-8f54-4755-815d-dd4e041065a3" (UID: "6b4600f4-8f54-4755-815d-dd4e041065a3"). InnerVolumeSpecName "kube-api-access-fttql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:08.150934 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:08.150899    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b4600f4-8f54-4755-815d-dd4e041065a3-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:08.150934 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:08.150931    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fttql\" (UniqueName: \"kubernetes.io/projected/6b4600f4-8f54-4755-815d-dd4e041065a3-kube-api-access-fttql\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:08.150934 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:08.150943    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b4600f4-8f54-4755-815d-dd4e041065a3-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:08.858046 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:08.858016    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edfdd95-qjp8v"
Nov 02 01:43:08.858534 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:08.858018    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edfdd95-qjp8v" event={"ID":"6b4600f4-8f54-4755-815d-dd4e041065a3","Type":"ContainerDied","Data":"676b905ef4e591c1e55ca15f1127d60d0696af564277629cfd9a87bf23660c3f"}
Nov 02 01:43:08.858534 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:08.858159    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="676b905ef4e591c1e55ca15f1127d60d0696af564277629cfd9a87bf23660c3f"
Nov 02 01:43:08.882314 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:08.880611    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5" event={"ID":"50ac256a-961a-4ff2-b454-06a201660154","Type":"ContainerStarted","Data":"20467e2fc98dc9d897326babfe7d0fa173b52a442c07a4b3bbdd338585e8d102"}
Nov 02 01:43:09.101771 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:09.101713    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5" podStartSLOduration=3.101697933 podStartE2EDuration="3.101697933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:09.085334133 +0000 UTC m=+1053.989016251" watchObservedRunningTime="2026-11-02 01:43:09.101697933 +0000 UTC m=+1054.005380046"
Nov 02 01:43:09.376317 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:09.367636    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:09.381317 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:09.379664    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 01:43:09.381317 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:09.379729    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5" podUID="50ac256a-961a-4ff2-b454-06a201660154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 01:43:10.318360 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:10.318328    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee317f08-6552p"]
Nov 02 01:43:10.318869 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:10.318853    5639 cpu_manager.go:414] "containerMap: removing container" podUID="6b4600f4-8f54-4755-815d-dd4e041065a3" containerName="revoke-keys"
Nov 02 01:43:10.318917 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:10.318871    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b4600f4-8f54-4755-815d-dd4e041065a3" containerName="revoke-keys"
Nov 02 01:43:10.319011 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:10.319002    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b4600f4-8f54-4755-815d-dd4e041065a3" containerName="revoke-keys"
Nov 02 01:43:10.322412 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:10.322391    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee317f08-6552p"
Nov 02 01:43:10.366748 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:10.366712    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 01:43:10.367031 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:10.367003    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5" podUID="50ac256a-961a-4ff2-b454-06a201660154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 01:43:10.390097 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:10.390061    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee317f08-6552p"]
Nov 02 01:43:10.400794 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:10.400758    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmwd5\" (UniqueName: \"kubernetes.io/projected/d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2-kube-api-access-nmwd5\") pod \"maas-api-revoke-sub-ee317f08-6552p\" (UID: \"d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee317f08-6552p"
Nov 02 01:43:10.401008 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:10.400988    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee317f08-6552p\" (UID: \"d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee317f08-6552p"
Nov 02 01:43:10.401084 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:10.401070    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee317f08-6552p\" (UID: \"d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee317f08-6552p"
Nov 02 01:43:10.435417 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:10.435378    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:43:10.435417 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:10.435393    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:43:10.435417 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:10.435386    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:43:10.435657 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:10.435461    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:43:10.502651 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:10.502617    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee317f08-6552p\" (UID: \"d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee317f08-6552p"
Nov 02 01:43:10.502849 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:10.502671    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmwd5\" (UniqueName: \"kubernetes.io/projected/d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2-kube-api-access-nmwd5\") pod \"maas-api-revoke-sub-ee317f08-6552p\" (UID: \"d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee317f08-6552p"
Nov 02 01:43:10.502849 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:10.502808    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee317f08-6552p\" (UID: \"d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee317f08-6552p"
Nov 02 01:43:10.503442 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:10.503418    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee317f08-6552p\" (UID: \"d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee317f08-6552p"
Nov 02 01:43:10.505164 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:10.505139    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee317f08-6552p\" (UID: \"d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee317f08-6552p"
Nov 02 01:43:10.590339 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:10.589642    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmwd5\" (UniqueName: \"kubernetes.io/projected/d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2-kube-api-access-nmwd5\") pod \"maas-api-revoke-sub-ee317f08-6552p\" (UID: \"d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee317f08-6552p"
Nov 02 01:43:10.637270 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:10.637236    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee317f08-6552p"
Nov 02 01:43:11.047846 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:11.047816    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee317f08-6552p"]
Nov 02 01:43:11.049440 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:43:11.049415    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd72a3c94_8d43_4a1f_b2f5_e74d2839a5f2.slice/crio-9e525accbc4f5fe7fc02bee0bde2bff77f764a35620406f4de13aa03d2f98818 WatchSource:0}: Error finding container 9e525accbc4f5fe7fc02bee0bde2bff77f764a35620406f4de13aa03d2f98818: Status 404 returned error can't find the container with id 9e525accbc4f5fe7fc02bee0bde2bff77f764a35620406f4de13aa03d2f98818
Nov 02 01:43:11.374313 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:11.365780    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 01:43:11.374313 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:11.365857    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5" podUID="50ac256a-961a-4ff2-b454-06a201660154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 01:43:11.907018 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:11.906982    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee317f08-6552p" event={"ID":"d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2","Type":"ContainerStarted","Data":"15d414964e543a06c555f6b5c02938a0cb8438147ac38cf41e8062fc11bc1d18"}
Nov 02 01:43:11.907214 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:11.907039    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee317f08-6552p" event={"ID":"d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2","Type":"ContainerStarted","Data":"9e525accbc4f5fe7fc02bee0bde2bff77f764a35620406f4de13aa03d2f98818"}
Nov 02 01:43:12.024336 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:12.024260    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee317f08-6552p" podStartSLOduration=2.024248085 podStartE2EDuration="2.024248085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:12.022750776 +0000 UTC m=+1056.926432891" watchObservedRunningTime="2026-11-02 01:43:12.024248085 +0000 UTC m=+1056.927930230"
Nov 02 01:43:12.369307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:12.365417    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 01:43:12.369307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:12.365478    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5" podUID="50ac256a-961a-4ff2-b454-06a201660154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 01:43:12.914705 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:12.914668    5639 generic.go:356] "Generic (PLEG): container finished" podID="d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2" containerID="15d414964e543a06c555f6b5c02938a0cb8438147ac38cf41e8062fc11bc1d18" exitCode=0
Nov 02 01:43:12.915138 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:12.914709    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee317f08-6552p" event={"ID":"d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2","Type":"ContainerDied","Data":"15d414964e543a06c555f6b5c02938a0cb8438147ac38cf41e8062fc11bc1d18"}
Nov 02 01:43:13.031857 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:13.031815    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:43:13.047004 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:13.046974    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:43:13.372208 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:13.371723    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 01:43:13.372208 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:13.371787    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5" podUID="50ac256a-961a-4ff2-b454-06a201660154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 01:43:13.686795 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:13.686691    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8a6a89a-khrmb"]
Nov 02 01:43:13.691179 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:13.691159    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a6a89a-khrmb"
Nov 02 01:43:13.743446 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:13.743224    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8a6a89a-khrmb"]
Nov 02 01:43:13.789163 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:13.789127    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8233db32-47lt9"]
Nov 02 01:43:13.793568 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:13.793545    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8233db32-47lt9"
Nov 02 01:43:13.841801 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:13.841764    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c88d48-ec32-4b53-b6e7-96e02b531e82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8a6a89a-khrmb\" (UID: \"40c88d48-ec32-4b53-b6e7-96e02b531e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a6a89a-khrmb"
Nov 02 01:43:13.841979 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:13.841928    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c88d48-ec32-4b53-b6e7-96e02b531e82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8a6a89a-khrmb\" (UID: \"40c88d48-ec32-4b53-b6e7-96e02b531e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a6a89a-khrmb"
Nov 02 01:43:13.842079 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:13.842042    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94njn\" (UniqueName: \"kubernetes.io/projected/40c88d48-ec32-4b53-b6e7-96e02b531e82-kube-api-access-94njn\") pod \"maas-api-revoke-sub-d8a6a89a-khrmb\" (UID: \"40c88d48-ec32-4b53-b6e7-96e02b531e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a6a89a-khrmb"
Nov 02 01:43:13.848723 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:13.848696    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8233db32-47lt9"]
Nov 02 01:43:13.938214 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:13.938127    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:43:13.942929 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:13.942895    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7h7b\" (UniqueName: \"kubernetes.io/projected/d77e8a8b-09b3-4122-9a17-2a1da2f5ab77-kube-api-access-h7h7b\") pod \"maas-api-revoke-sub-8233db32-47lt9\" (UID: \"d77e8a8b-09b3-4122-9a17-2a1da2f5ab77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8233db32-47lt9"
Nov 02 01:43:13.943033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:13.942952    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c88d48-ec32-4b53-b6e7-96e02b531e82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8a6a89a-khrmb\" (UID: \"40c88d48-ec32-4b53-b6e7-96e02b531e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a6a89a-khrmb"
Nov 02 01:43:13.943084 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:13.943035    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d77e8a8b-09b3-4122-9a17-2a1da2f5ab77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8233db32-47lt9\" (UID: \"d77e8a8b-09b3-4122-9a17-2a1da2f5ab77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8233db32-47lt9"
Nov 02 01:43:13.943172 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:13.943154    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94njn\" (UniqueName: \"kubernetes.io/projected/40c88d48-ec32-4b53-b6e7-96e02b531e82-kube-api-access-94njn\") pod \"maas-api-revoke-sub-d8a6a89a-khrmb\" (UID: \"40c88d48-ec32-4b53-b6e7-96e02b531e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a6a89a-khrmb"
Nov 02 01:43:13.943255 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:13.943240    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c88d48-ec32-4b53-b6e7-96e02b531e82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8a6a89a-khrmb\" (UID: \"40c88d48-ec32-4b53-b6e7-96e02b531e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a6a89a-khrmb"
Nov 02 01:43:13.943391 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:13.943372    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d77e8a8b-09b3-4122-9a17-2a1da2f5ab77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8233db32-47lt9\" (UID: \"d77e8a8b-09b3-4122-9a17-2a1da2f5ab77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8233db32-47lt9"
Nov 02 01:43:13.943884 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:13.943862    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c88d48-ec32-4b53-b6e7-96e02b531e82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8a6a89a-khrmb\" (UID: \"40c88d48-ec32-4b53-b6e7-96e02b531e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a6a89a-khrmb"
Nov 02 01:43:13.945627 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:13.945607    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c88d48-ec32-4b53-b6e7-96e02b531e82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8a6a89a-khrmb\" (UID: \"40c88d48-ec32-4b53-b6e7-96e02b531e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a6a89a-khrmb"
Nov 02 01:43:14.006212 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.006174    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94njn\" (UniqueName: \"kubernetes.io/projected/40c88d48-ec32-4b53-b6e7-96e02b531e82-kube-api-access-94njn\") pod \"maas-api-revoke-sub-d8a6a89a-khrmb\" (UID: \"40c88d48-ec32-4b53-b6e7-96e02b531e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a6a89a-khrmb"
Nov 02 01:43:14.044649 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.044613    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d77e8a8b-09b3-4122-9a17-2a1da2f5ab77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8233db32-47lt9\" (UID: \"d77e8a8b-09b3-4122-9a17-2a1da2f5ab77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8233db32-47lt9"
Nov 02 01:43:14.044829 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.044815    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d77e8a8b-09b3-4122-9a17-2a1da2f5ab77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8233db32-47lt9\" (UID: \"d77e8a8b-09b3-4122-9a17-2a1da2f5ab77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8233db32-47lt9"
Nov 02 01:43:14.045184 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.044915    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7h7b\" (UniqueName: \"kubernetes.io/projected/d77e8a8b-09b3-4122-9a17-2a1da2f5ab77-kube-api-access-h7h7b\") pod \"maas-api-revoke-sub-8233db32-47lt9\" (UID: \"d77e8a8b-09b3-4122-9a17-2a1da2f5ab77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8233db32-47lt9"
Nov 02 01:43:14.045638 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.045617    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d77e8a8b-09b3-4122-9a17-2a1da2f5ab77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8233db32-47lt9\" (UID: \"d77e8a8b-09b3-4122-9a17-2a1da2f5ab77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8233db32-47lt9"
Nov 02 01:43:14.047196 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.047173    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d77e8a8b-09b3-4122-9a17-2a1da2f5ab77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8233db32-47lt9\" (UID: \"d77e8a8b-09b3-4122-9a17-2a1da2f5ab77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8233db32-47lt9"
Nov 02 01:43:14.082485 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.082448    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee317f08-6552p"
Nov 02 01:43:14.106189 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.106155    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7h7b\" (UniqueName: \"kubernetes.io/projected/d77e8a8b-09b3-4122-9a17-2a1da2f5ab77-kube-api-access-h7h7b\") pod \"maas-api-revoke-sub-8233db32-47lt9\" (UID: \"d77e8a8b-09b3-4122-9a17-2a1da2f5ab77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8233db32-47lt9"
Nov 02 01:43:14.107767 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.107746    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8233db32-47lt9"
Nov 02 01:43:14.246958 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.246834    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2-maas-api-xks-ca\") pod \"d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2\" (UID: \"d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2\") "
Nov 02 01:43:14.247138 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.247030    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2-maas-api-openshift-ca\") pod \"d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2\" (UID: \"d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2\") "
Nov 02 01:43:14.247138 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.247088    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2-kube-api-access-nmwd5\" (UniqueName: \"kubernetes.io/projected/d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2-kube-api-access-nmwd5\") pod \"d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2\" (UID: \"d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2\") "
Nov 02 01:43:14.247407 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.247383    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2-maas-api-openshift-ca" pod "d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2" (UID: "d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:14.247570 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.247551    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:14.249215 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.249192    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2-maas-api-xks-ca" pod "d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2" (UID: "d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:14.249353 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.249328    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2-kube-api-access-nmwd5" pod "d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2" (UID: "d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2"). InnerVolumeSpecName "kube-api-access-nmwd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:14.305635 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.305597    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a6a89a-khrmb"
Nov 02 01:43:14.349344 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.349270    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmwd5\" (UniqueName: \"kubernetes.io/projected/d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2-kube-api-access-nmwd5\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:14.349344 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.349347    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:14.370728 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.370701    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 01:43:14.370832 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.370759    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5" podUID="50ac256a-961a-4ff2-b454-06a201660154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 01:43:14.485196 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.485170    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8233db32-47lt9"]
Nov 02 01:43:14.487239 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:43:14.487207    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77e8a8b_09b3_4122_9a17_2a1da2f5ab77.slice/crio-1fcb9bbab1e91a77be188a29024063c092df420023b0b87c6989d69380adaa1f WatchSource:0}: Error finding container 1fcb9bbab1e91a77be188a29024063c092df420023b0b87c6989d69380adaa1f: Status 404 returned error can't find the container with id 1fcb9bbab1e91a77be188a29024063c092df420023b0b87c6989d69380adaa1f
Nov 02 01:43:14.690026 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.690000    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8a6a89a-khrmb"]
Nov 02 01:43:14.691109 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:43:14.691079    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c88d48_ec32_4b53_b6e7_96e02b531e82.slice/crio-f4e40458fb1673a62cc0f827e09d23dbc4b0feb20139fa2d6623ce7b88cc5c52 WatchSource:0}: Error finding container f4e40458fb1673a62cc0f827e09d23dbc4b0feb20139fa2d6623ce7b88cc5c52: Status 404 returned error can't find the container with id f4e40458fb1673a62cc0f827e09d23dbc4b0feb20139fa2d6623ce7b88cc5c52
Nov 02 01:43:14.930200 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.930166    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a6a89a-khrmb" event={"ID":"40c88d48-ec32-4b53-b6e7-96e02b531e82","Type":"ContainerStarted","Data":"f4e40458fb1673a62cc0f827e09d23dbc4b0feb20139fa2d6623ce7b88cc5c52"}
Nov 02 01:43:14.931604 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.931584    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee317f08-6552p"
Nov 02 01:43:14.931604 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.931599    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee317f08-6552p" event={"ID":"d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2","Type":"ContainerDied","Data":"9e525accbc4f5fe7fc02bee0bde2bff77f764a35620406f4de13aa03d2f98818"}
Nov 02 01:43:14.931821 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.931643    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e525accbc4f5fe7fc02bee0bde2bff77f764a35620406f4de13aa03d2f98818"
Nov 02 01:43:14.933466 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.933429    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8233db32-47lt9" event={"ID":"d77e8a8b-09b3-4122-9a17-2a1da2f5ab77","Type":"ContainerStarted","Data":"9e5cbdb374459051e3684c5a21231ef03076a128c18a0abe830884144d8ba601"}
Nov 02 01:43:14.933574 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:14.933488    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8233db32-47lt9" event={"ID":"d77e8a8b-09b3-4122-9a17-2a1da2f5ab77","Type":"ContainerStarted","Data":"1fcb9bbab1e91a77be188a29024063c092df420023b0b87c6989d69380adaa1f"}
Nov 02 01:43:15.048541 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:15.048441    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8233db32-47lt9" podStartSLOduration=2.04842494 podStartE2EDuration="2.04842494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:15.047657308 +0000 UTC m=+1059.951339423" watchObservedRunningTime="2026-11-02 01:43:15.04842494 +0000 UTC m=+1059.952107054"
Nov 02 01:43:15.369875 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:15.369843    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:15.370177 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:15.370155    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:15.371100 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:15.371072    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 01:43:15.941500 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:15.941408    5639 generic.go:356] "Generic (PLEG): container finished" podID="d77e8a8b-09b3-4122-9a17-2a1da2f5ab77" containerID="9e5cbdb374459051e3684c5a21231ef03076a128c18a0abe830884144d8ba601" exitCode=0
Nov 02 01:43:15.941500 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:15.941478    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8233db32-47lt9" event={"ID":"d77e8a8b-09b3-4122-9a17-2a1da2f5ab77","Type":"ContainerDied","Data":"9e5cbdb374459051e3684c5a21231ef03076a128c18a0abe830884144d8ba601"}
Nov 02 01:43:15.943135 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:15.943114    5639 generic.go:356] "Generic (PLEG): container finished" podID="40c88d48-ec32-4b53-b6e7-96e02b531e82" containerID="9f3f53b5da29d89a03b7a85b299aeb732627e9d3e36727b3bc2aae20db4ad467" exitCode=0
Nov 02 01:43:15.943300 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:15.943171    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a6a89a-khrmb" event={"ID":"40c88d48-ec32-4b53-b6e7-96e02b531e82","Type":"ContainerDied","Data":"9f3f53b5da29d89a03b7a85b299aeb732627e9d3e36727b3bc2aae20db4ad467"}
Nov 02 01:43:17.144381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:17.144357    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8233db32-47lt9"
Nov 02 01:43:17.147715 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:17.147692    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a6a89a-khrmb"
Nov 02 01:43:17.284479 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:17.284383    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d77e8a8b-09b3-4122-9a17-2a1da2f5ab77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d77e8a8b-09b3-4122-9a17-2a1da2f5ab77-maas-api-openshift-ca\") pod \"d77e8a8b-09b3-4122-9a17-2a1da2f5ab77\" (UID: \"d77e8a8b-09b3-4122-9a17-2a1da2f5ab77\") "
Nov 02 01:43:17.284642 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:17.284515    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d77e8a8b-09b3-4122-9a17-2a1da2f5ab77-kube-api-access-h7h7b\" (UniqueName: \"kubernetes.io/projected/d77e8a8b-09b3-4122-9a17-2a1da2f5ab77-kube-api-access-h7h7b\") pod \"d77e8a8b-09b3-4122-9a17-2a1da2f5ab77\" (UID: \"d77e8a8b-09b3-4122-9a17-2a1da2f5ab77\") "
Nov 02 01:43:17.284642 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:17.284599    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40c88d48-ec32-4b53-b6e7-96e02b531e82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c88d48-ec32-4b53-b6e7-96e02b531e82-maas-api-openshift-ca\") pod \"40c88d48-ec32-4b53-b6e7-96e02b531e82\" (UID: \"40c88d48-ec32-4b53-b6e7-96e02b531e82\") "
Nov 02 01:43:17.284766 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:17.284675    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c88d48-ec32-4b53-b6e7-96e02b531e82-kube-api-access-94njn\" (UniqueName: \"kubernetes.io/projected/40c88d48-ec32-4b53-b6e7-96e02b531e82-kube-api-access-94njn\") pod \"40c88d48-ec32-4b53-b6e7-96e02b531e82\" (UID: \"40c88d48-ec32-4b53-b6e7-96e02b531e82\") "
Nov 02 01:43:17.284766 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:17.284752    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40c88d48-ec32-4b53-b6e7-96e02b531e82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c88d48-ec32-4b53-b6e7-96e02b531e82-maas-api-xks-ca\") pod \"40c88d48-ec32-4b53-b6e7-96e02b531e82\" (UID: \"40c88d48-ec32-4b53-b6e7-96e02b531e82\") "
Nov 02 01:43:17.284875 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:17.284819    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d77e8a8b-09b3-4122-9a17-2a1da2f5ab77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d77e8a8b-09b3-4122-9a17-2a1da2f5ab77-maas-api-xks-ca\") pod \"d77e8a8b-09b3-4122-9a17-2a1da2f5ab77\" (UID: \"d77e8a8b-09b3-4122-9a17-2a1da2f5ab77\") "
Nov 02 01:43:17.284933 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:17.284905    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d77e8a8b-09b3-4122-9a17-2a1da2f5ab77-maas-api-openshift-ca" pod "d77e8a8b-09b3-4122-9a17-2a1da2f5ab77" (UID: "d77e8a8b-09b3-4122-9a17-2a1da2f5ab77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:17.285047 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:17.285025    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c88d48-ec32-4b53-b6e7-96e02b531e82-maas-api-openshift-ca" pod "40c88d48-ec32-4b53-b6e7-96e02b531e82" (UID: "40c88d48-ec32-4b53-b6e7-96e02b531e82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:17.285325 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:17.285299    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c88d48-ec32-4b53-b6e7-96e02b531e82-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:17.285459 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:17.285329    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d77e8a8b-09b3-4122-9a17-2a1da2f5ab77-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:17.287170 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:17.287132    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d77e8a8b-09b3-4122-9a17-2a1da2f5ab77-maas-api-xks-ca" pod "d77e8a8b-09b3-4122-9a17-2a1da2f5ab77" (UID: "d77e8a8b-09b3-4122-9a17-2a1da2f5ab77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:17.287547 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:17.287516    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c88d48-ec32-4b53-b6e7-96e02b531e82-kube-api-access-94njn" pod "40c88d48-ec32-4b53-b6e7-96e02b531e82" (UID: "40c88d48-ec32-4b53-b6e7-96e02b531e82"). InnerVolumeSpecName "kube-api-access-94njn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:17.287658 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:17.287643    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77e8a8b-09b3-4122-9a17-2a1da2f5ab77-kube-api-access-h7h7b" pod "d77e8a8b-09b3-4122-9a17-2a1da2f5ab77" (UID: "d77e8a8b-09b3-4122-9a17-2a1da2f5ab77"). InnerVolumeSpecName "kube-api-access-h7h7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:17.287741 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:17.287664    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c88d48-ec32-4b53-b6e7-96e02b531e82-maas-api-xks-ca" pod "40c88d48-ec32-4b53-b6e7-96e02b531e82" (UID: "40c88d48-ec32-4b53-b6e7-96e02b531e82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:17.386775 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:17.386746    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94njn\" (UniqueName: \"kubernetes.io/projected/40c88d48-ec32-4b53-b6e7-96e02b531e82-kube-api-access-94njn\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:17.386775 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:17.386774    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c88d48-ec32-4b53-b6e7-96e02b531e82-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:17.386972 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:17.386787    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d77e8a8b-09b3-4122-9a17-2a1da2f5ab77-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:17.386972 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:17.386800    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7h7b\" (UniqueName: \"kubernetes.io/projected/d77e8a8b-09b3-4122-9a17-2a1da2f5ab77-kube-api-access-h7h7b\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:17.964147 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:17.964111    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a6a89a-khrmb"
Nov 02 01:43:17.964374 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:17.964147    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a6a89a-khrmb" event={"ID":"40c88d48-ec32-4b53-b6e7-96e02b531e82","Type":"ContainerDied","Data":"f4e40458fb1673a62cc0f827e09d23dbc4b0feb20139fa2d6623ce7b88cc5c52"}
Nov 02 01:43:17.964374 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:17.964208    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4e40458fb1673a62cc0f827e09d23dbc4b0feb20139fa2d6623ce7b88cc5c52"
Nov 02 01:43:17.965729 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:17.965703    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8233db32-47lt9"
Nov 02 01:43:17.965861 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:17.965798    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8233db32-47lt9" event={"ID":"d77e8a8b-09b3-4122-9a17-2a1da2f5ab77","Type":"ContainerDied","Data":"1fcb9bbab1e91a77be188a29024063c092df420023b0b87c6989d69380adaa1f"}
Nov 02 01:43:17.965861 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:17.965838    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fcb9bbab1e91a77be188a29024063c092df420023b0b87c6989d69380adaa1f"
Nov 02 01:43:18.802852 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:18.802810    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:43:20.628890 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:20.628857    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68278e7c-8xtqp"]
Nov 02 01:43:20.630234 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:20.630204    5639 cpu_manager.go:414] "containerMap: removing container" podUID="d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2" containerName="revoke-keys"
Nov 02 01:43:20.638327 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:20.638263    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2" containerName="revoke-keys"
Nov 02 01:43:20.638501 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:20.638488    5639 cpu_manager.go:414] "containerMap: removing container" podUID="d77e8a8b-09b3-4122-9a17-2a1da2f5ab77" containerName="revoke-keys"
Nov 02 01:43:20.638608 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:20.638596    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d77e8a8b-09b3-4122-9a17-2a1da2f5ab77" containerName="revoke-keys"
Nov 02 01:43:20.638733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:20.638722    5639 cpu_manager.go:414] "containerMap: removing container" podUID="40c88d48-ec32-4b53-b6e7-96e02b531e82" containerName="revoke-keys"
Nov 02 01:43:20.638809 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:20.638798    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c88d48-ec32-4b53-b6e7-96e02b531e82" containerName="revoke-keys"
Nov 02 01:43:20.639112 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:20.639093    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2" containerName="revoke-keys"
Nov 02 01:43:20.639223 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:20.639212    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="d77e8a8b-09b3-4122-9a17-2a1da2f5ab77" containerName="revoke-keys"
Nov 02 01:43:20.639369 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:20.639358    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c88d48-ec32-4b53-b6e7-96e02b531e82" containerName="revoke-keys"
Nov 02 01:43:20.655334 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:20.654957    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68278e7c-8xtqp"
Nov 02 01:43:20.703491 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:20.703457    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68278e7c-8xtqp"]
Nov 02 01:43:20.753398 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:20.753365    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:43:20.754297 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:20.754253    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:43:20.756427 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:20.754420    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:43:20.756427 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:20.754446    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:43:20.854310 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:20.854202    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpgnj\" (UniqueName: \"kubernetes.io/projected/9c19baea-9ea4-4e66-99a1-056a3c93ad99-kube-api-access-cpgnj\") pod \"maas-api-revoke-sub-68278e7c-8xtqp\" (UID: \"9c19baea-9ea4-4e66-99a1-056a3c93ad99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68278e7c-8xtqp"
Nov 02 01:43:20.854310 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:20.854261    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c19baea-9ea4-4e66-99a1-056a3c93ad99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68278e7c-8xtqp\" (UID: \"9c19baea-9ea4-4e66-99a1-056a3c93ad99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68278e7c-8xtqp"
Nov 02 01:43:20.854581 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:20.854353    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c19baea-9ea4-4e66-99a1-056a3c93ad99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68278e7c-8xtqp\" (UID: \"9c19baea-9ea4-4e66-99a1-056a3c93ad99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68278e7c-8xtqp"
Nov 02 01:43:20.955172 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:20.955073    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpgnj\" (UniqueName: \"kubernetes.io/projected/9c19baea-9ea4-4e66-99a1-056a3c93ad99-kube-api-access-cpgnj\") pod \"maas-api-revoke-sub-68278e7c-8xtqp\" (UID: \"9c19baea-9ea4-4e66-99a1-056a3c93ad99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68278e7c-8xtqp"
Nov 02 01:43:20.955172 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:20.955120    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c19baea-9ea4-4e66-99a1-056a3c93ad99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68278e7c-8xtqp\" (UID: \"9c19baea-9ea4-4e66-99a1-056a3c93ad99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68278e7c-8xtqp"
Nov 02 01:43:20.955172 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:20.955142    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c19baea-9ea4-4e66-99a1-056a3c93ad99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68278e7c-8xtqp\" (UID: \"9c19baea-9ea4-4e66-99a1-056a3c93ad99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68278e7c-8xtqp"
Nov 02 01:43:20.955836 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:20.955814    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c19baea-9ea4-4e66-99a1-056a3c93ad99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68278e7c-8xtqp\" (UID: \"9c19baea-9ea4-4e66-99a1-056a3c93ad99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68278e7c-8xtqp"
Nov 02 01:43:20.958206 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:20.958176    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c19baea-9ea4-4e66-99a1-056a3c93ad99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68278e7c-8xtqp\" (UID: \"9c19baea-9ea4-4e66-99a1-056a3c93ad99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68278e7c-8xtqp"
Nov 02 01:43:21.032895 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:21.032859    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpgnj\" (UniqueName: \"kubernetes.io/projected/9c19baea-9ea4-4e66-99a1-056a3c93ad99-kube-api-access-cpgnj\") pod \"maas-api-revoke-sub-68278e7c-8xtqp\" (UID: \"9c19baea-9ea4-4e66-99a1-056a3c93ad99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68278e7c-8xtqp"
Nov 02 01:43:21.282596 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:21.282512    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68278e7c-8xtqp"
Nov 02 01:43:21.667755 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:21.667725    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68278e7c-8xtqp"]
Nov 02 01:43:21.668994 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:43:21.668968    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c19baea_9ea4_4e66_99a1_056a3c93ad99.slice/crio-80b41718d857f52f80feac9869507e393e7bab309f13bfde1721cb24093c001d WatchSource:0}: Error finding container 80b41718d857f52f80feac9869507e393e7bab309f13bfde1721cb24093c001d: Status 404 returned error can't find the container with id 80b41718d857f52f80feac9869507e393e7bab309f13bfde1721cb24093c001d
Nov 02 01:43:22.045885 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:22.045845    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68278e7c-8xtqp" event={"ID":"9c19baea-9ea4-4e66-99a1-056a3c93ad99","Type":"ContainerStarted","Data":"899cd3e1f5d00a8e120afd2e4130d074f693f7fce584a84555d68f7b7e973bf7"}
Nov 02 01:43:22.046054 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:22.045901    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68278e7c-8xtqp" event={"ID":"9c19baea-9ea4-4e66-99a1-056a3c93ad99","Type":"ContainerStarted","Data":"80b41718d857f52f80feac9869507e393e7bab309f13bfde1721cb24093c001d"}
Nov 02 01:43:22.160942 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:22.160741    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68278e7c-8xtqp" podStartSLOduration=2.16072197 podStartE2EDuration="2.16072197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:22.160457381 +0000 UTC m=+1067.064139508" watchObservedRunningTime="2026-11-02 01:43:22.16072197 +0000 UTC m=+1067.064404085"
Nov 02 01:43:23.055084 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:23.055049    5639 generic.go:356] "Generic (PLEG): container finished" podID="9c19baea-9ea4-4e66-99a1-056a3c93ad99" containerID="899cd3e1f5d00a8e120afd2e4130d074f693f7fce584a84555d68f7b7e973bf7" exitCode=0
Nov 02 01:43:23.055505 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:23.055110    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68278e7c-8xtqp" event={"ID":"9c19baea-9ea4-4e66-99a1-056a3c93ad99","Type":"ContainerDied","Data":"899cd3e1f5d00a8e120afd2e4130d074f693f7fce584a84555d68f7b7e973bf7"}
Nov 02 01:43:24.214781 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:24.214753    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68278e7c-8xtqp"
Nov 02 01:43:24.295213 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:24.295182    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c19baea-9ea4-4e66-99a1-056a3c93ad99-kube-api-access-cpgnj\" (UniqueName: \"kubernetes.io/projected/9c19baea-9ea4-4e66-99a1-056a3c93ad99-kube-api-access-cpgnj\") pod \"9c19baea-9ea4-4e66-99a1-056a3c93ad99\" (UID: \"9c19baea-9ea4-4e66-99a1-056a3c93ad99\") "
Nov 02 01:43:24.295688 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:24.295257    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c19baea-9ea4-4e66-99a1-056a3c93ad99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c19baea-9ea4-4e66-99a1-056a3c93ad99-maas-api-openshift-ca\") pod \"9c19baea-9ea4-4e66-99a1-056a3c93ad99\" (UID: \"9c19baea-9ea4-4e66-99a1-056a3c93ad99\") "
Nov 02 01:43:24.295688 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:24.295393    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c19baea-9ea4-4e66-99a1-056a3c93ad99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c19baea-9ea4-4e66-99a1-056a3c93ad99-maas-api-xks-ca\") pod \"9c19baea-9ea4-4e66-99a1-056a3c93ad99\" (UID: \"9c19baea-9ea4-4e66-99a1-056a3c93ad99\") "
Nov 02 01:43:24.295688 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:24.295675    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c19baea-9ea4-4e66-99a1-056a3c93ad99-maas-api-openshift-ca" pod "9c19baea-9ea4-4e66-99a1-056a3c93ad99" (UID: "9c19baea-9ea4-4e66-99a1-056a3c93ad99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:24.297552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:24.297524    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c19baea-9ea4-4e66-99a1-056a3c93ad99-maas-api-xks-ca" pod "9c19baea-9ea4-4e66-99a1-056a3c93ad99" (UID: "9c19baea-9ea4-4e66-99a1-056a3c93ad99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:24.297634 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:24.297549    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c19baea-9ea4-4e66-99a1-056a3c93ad99-kube-api-access-cpgnj" pod "9c19baea-9ea4-4e66-99a1-056a3c93ad99" (UID: "9c19baea-9ea4-4e66-99a1-056a3c93ad99"). InnerVolumeSpecName "kube-api-access-cpgnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:24.396336 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:24.396305    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpgnj\" (UniqueName: \"kubernetes.io/projected/9c19baea-9ea4-4e66-99a1-056a3c93ad99-kube-api-access-cpgnj\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:24.396336 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:24.396330    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c19baea-9ea4-4e66-99a1-056a3c93ad99-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:24.396336 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:24.396340    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c19baea-9ea4-4e66-99a1-056a3c93ad99-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:25.069049 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:25.069018    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68278e7c-8xtqp"
Nov 02 01:43:25.069049 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:25.069040    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68278e7c-8xtqp" event={"ID":"9c19baea-9ea4-4e66-99a1-056a3c93ad99","Type":"ContainerDied","Data":"80b41718d857f52f80feac9869507e393e7bab309f13bfde1721cb24093c001d"}
Nov 02 01:43:25.069303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:25.069094    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80b41718d857f52f80feac9869507e393e7bab309f13bfde1721cb24093c001d"
Nov 02 01:43:26.326928 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:26.326896    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8408d89d-f6p4s"]
Nov 02 01:43:26.327541 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:26.327525    5639 cpu_manager.go:414] "containerMap: removing container" podUID="9c19baea-9ea4-4e66-99a1-056a3c93ad99" containerName="revoke-keys"
Nov 02 01:43:26.327599 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:26.327543    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c19baea-9ea4-4e66-99a1-056a3c93ad99" containerName="revoke-keys"
Nov 02 01:43:26.327662 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:26.327653    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c19baea-9ea4-4e66-99a1-056a3c93ad99" containerName="revoke-keys"
Nov 02 01:43:26.331407 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:26.331386    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8408d89d-f6p4s"
Nov 02 01:43:26.387378 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:26.387346    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8408d89d-f6p4s"]
Nov 02 01:43:26.419345 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:26.419304    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6578a06-3971-4811-9505-ee4eaf1a5627-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8408d89d-f6p4s\" (UID: \"c6578a06-3971-4811-9505-ee4eaf1a5627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8408d89d-f6p4s"
Nov 02 01:43:26.419512 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:26.419430    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55crl\" (UniqueName: \"kubernetes.io/projected/c6578a06-3971-4811-9505-ee4eaf1a5627-kube-api-access-55crl\") pod \"maas-api-revoke-sub-8408d89d-f6p4s\" (UID: \"c6578a06-3971-4811-9505-ee4eaf1a5627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8408d89d-f6p4s"
Nov 02 01:43:26.419512 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:26.419491    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6578a06-3971-4811-9505-ee4eaf1a5627-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8408d89d-f6p4s\" (UID: \"c6578a06-3971-4811-9505-ee4eaf1a5627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8408d89d-f6p4s"
Nov 02 01:43:26.432819 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:26.432791    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:43:26.432998 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:26.432887    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:43:26.432998 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:26.432930    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:43:26.433164 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:26.433149    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:43:26.521061 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:26.521022    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55crl\" (UniqueName: \"kubernetes.io/projected/c6578a06-3971-4811-9505-ee4eaf1a5627-kube-api-access-55crl\") pod \"maas-api-revoke-sub-8408d89d-f6p4s\" (UID: \"c6578a06-3971-4811-9505-ee4eaf1a5627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8408d89d-f6p4s"
Nov 02 01:43:26.521257 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:26.521104    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6578a06-3971-4811-9505-ee4eaf1a5627-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8408d89d-f6p4s\" (UID: \"c6578a06-3971-4811-9505-ee4eaf1a5627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8408d89d-f6p4s"
Nov 02 01:43:26.521257 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:26.521183    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6578a06-3971-4811-9505-ee4eaf1a5627-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8408d89d-f6p4s\" (UID: \"c6578a06-3971-4811-9505-ee4eaf1a5627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8408d89d-f6p4s"
Nov 02 01:43:26.521762 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:26.521741    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6578a06-3971-4811-9505-ee4eaf1a5627-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8408d89d-f6p4s\" (UID: \"c6578a06-3971-4811-9505-ee4eaf1a5627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8408d89d-f6p4s"
Nov 02 01:43:26.523943 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:26.523920    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6578a06-3971-4811-9505-ee4eaf1a5627-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8408d89d-f6p4s\" (UID: \"c6578a06-3971-4811-9505-ee4eaf1a5627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8408d89d-f6p4s"
Nov 02 01:43:26.582387 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:26.582363    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55crl\" (UniqueName: \"kubernetes.io/projected/c6578a06-3971-4811-9505-ee4eaf1a5627-kube-api-access-55crl\") pod \"maas-api-revoke-sub-8408d89d-f6p4s\" (UID: \"c6578a06-3971-4811-9505-ee4eaf1a5627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8408d89d-f6p4s"
Nov 02 01:43:26.648601 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:26.648570    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8408d89d-f6p4s"
Nov 02 01:43:27.027300 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:27.027254    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8408d89d-f6p4s"]
Nov 02 01:43:27.028544 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:43:27.028511    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6578a06_3971_4811_9505_ee4eaf1a5627.slice/crio-f2706eb90f6427816d69258aa46ad5a5e9ebcb98a1dc4617a5843974c8f00824 WatchSource:0}: Error finding container f2706eb90f6427816d69258aa46ad5a5e9ebcb98a1dc4617a5843974c8f00824: Status 404 returned error can't find the container with id f2706eb90f6427816d69258aa46ad5a5e9ebcb98a1dc4617a5843974c8f00824
Nov 02 01:43:27.094340 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:27.094258    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8408d89d-f6p4s" event={"ID":"c6578a06-3971-4811-9505-ee4eaf1a5627","Type":"ContainerStarted","Data":"f2706eb90f6427816d69258aa46ad5a5e9ebcb98a1dc4617a5843974c8f00824"}
Nov 02 01:43:27.421421 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:43:27.421384    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:43:28.102461 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:28.102418    5639 generic.go:356] "Generic (PLEG): container finished" podID="c6578a06-3971-4811-9505-ee4eaf1a5627" containerID="b48ae12da37af2f5f102152c5339fb41edd7e4a81cec92b276f919850a57092a" exitCode=0
Nov 02 01:43:28.102649 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:28.102500    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8408d89d-f6p4s" event={"ID":"c6578a06-3971-4811-9505-ee4eaf1a5627","Type":"ContainerDied","Data":"b48ae12da37af2f5f102152c5339fb41edd7e4a81cec92b276f919850a57092a"}
Nov 02 01:43:29.068131 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:29.068095    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-278149b1-fk6vc"]
Nov 02 01:43:29.072123 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:29.072106    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-278149b1-fk6vc"
Nov 02 01:43:29.130539 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:29.130505    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-278149b1-fk6vc"]
Nov 02 01:43:29.150685 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:29.150645    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkvql\" (UniqueName: \"kubernetes.io/projected/8fd58675-c64e-4a5f-bb11-a87dd9f49852-kube-api-access-tkvql\") pod \"maas-api-revoke-sub-278149b1-fk6vc\" (UID: \"8fd58675-c64e-4a5f-bb11-a87dd9f49852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-278149b1-fk6vc"
Nov 02 01:43:29.150877 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:29.150725    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fd58675-c64e-4a5f-bb11-a87dd9f49852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-278149b1-fk6vc\" (UID: \"8fd58675-c64e-4a5f-bb11-a87dd9f49852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-278149b1-fk6vc"
Nov 02 01:43:29.150877 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:29.150841    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fd58675-c64e-4a5f-bb11-a87dd9f49852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-278149b1-fk6vc\" (UID: \"8fd58675-c64e-4a5f-bb11-a87dd9f49852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-278149b1-fk6vc"
Nov 02 01:43:29.251733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:29.251701    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkvql\" (UniqueName: \"kubernetes.io/projected/8fd58675-c64e-4a5f-bb11-a87dd9f49852-kube-api-access-tkvql\") pod \"maas-api-revoke-sub-278149b1-fk6vc\" (UID: \"8fd58675-c64e-4a5f-bb11-a87dd9f49852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-278149b1-fk6vc"
Nov 02 01:43:29.251899 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:29.251740    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fd58675-c64e-4a5f-bb11-a87dd9f49852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-278149b1-fk6vc\" (UID: \"8fd58675-c64e-4a5f-bb11-a87dd9f49852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-278149b1-fk6vc"
Nov 02 01:43:29.251899 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:29.251825    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fd58675-c64e-4a5f-bb11-a87dd9f49852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-278149b1-fk6vc\" (UID: \"8fd58675-c64e-4a5f-bb11-a87dd9f49852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-278149b1-fk6vc"
Nov 02 01:43:29.252396 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:29.252375    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fd58675-c64e-4a5f-bb11-a87dd9f49852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-278149b1-fk6vc\" (UID: \"8fd58675-c64e-4a5f-bb11-a87dd9f49852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-278149b1-fk6vc"
Nov 02 01:43:29.254443 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:29.254418    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fd58675-c64e-4a5f-bb11-a87dd9f49852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-278149b1-fk6vc\" (UID: \"8fd58675-c64e-4a5f-bb11-a87dd9f49852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-278149b1-fk6vc"
Nov 02 01:43:29.274067 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:29.274045    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8408d89d-f6p4s"
Nov 02 01:43:29.312943 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:29.312907    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkvql\" (UniqueName: \"kubernetes.io/projected/8fd58675-c64e-4a5f-bb11-a87dd9f49852-kube-api-access-tkvql\") pod \"maas-api-revoke-sub-278149b1-fk6vc\" (UID: \"8fd58675-c64e-4a5f-bb11-a87dd9f49852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-278149b1-fk6vc"
Nov 02 01:43:29.352263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:29.352233    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6578a06-3971-4811-9505-ee4eaf1a5627-kube-api-access-55crl\" (UniqueName: \"kubernetes.io/projected/c6578a06-3971-4811-9505-ee4eaf1a5627-kube-api-access-55crl\") pod \"c6578a06-3971-4811-9505-ee4eaf1a5627\" (UID: \"c6578a06-3971-4811-9505-ee4eaf1a5627\") "
Nov 02 01:43:29.352490 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:29.352394    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6578a06-3971-4811-9505-ee4eaf1a5627-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6578a06-3971-4811-9505-ee4eaf1a5627-maas-api-xks-ca\") pod \"c6578a06-3971-4811-9505-ee4eaf1a5627\" (UID: \"c6578a06-3971-4811-9505-ee4eaf1a5627\") "
Nov 02 01:43:29.352490 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:29.352431    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6578a06-3971-4811-9505-ee4eaf1a5627-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6578a06-3971-4811-9505-ee4eaf1a5627-maas-api-openshift-ca\") pod \"c6578a06-3971-4811-9505-ee4eaf1a5627\" (UID: \"c6578a06-3971-4811-9505-ee4eaf1a5627\") "
Nov 02 01:43:29.352850 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:29.352822    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6578a06-3971-4811-9505-ee4eaf1a5627-maas-api-openshift-ca" pod "c6578a06-3971-4811-9505-ee4eaf1a5627" (UID: "c6578a06-3971-4811-9505-ee4eaf1a5627"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:29.353002 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:29.352985    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6578a06-3971-4811-9505-ee4eaf1a5627-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:29.354573 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:29.354545    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6578a06-3971-4811-9505-ee4eaf1a5627-kube-api-access-55crl" pod "c6578a06-3971-4811-9505-ee4eaf1a5627" (UID: "c6578a06-3971-4811-9505-ee4eaf1a5627"). InnerVolumeSpecName "kube-api-access-55crl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:29.354704 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:29.354670    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6578a06-3971-4811-9505-ee4eaf1a5627-maas-api-xks-ca" pod "c6578a06-3971-4811-9505-ee4eaf1a5627" (UID: "c6578a06-3971-4811-9505-ee4eaf1a5627"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:29.390070 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:29.389897    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-278149b1-fk6vc"
Nov 02 01:43:29.454985 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:29.454939    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55crl\" (UniqueName: \"kubernetes.io/projected/c6578a06-3971-4811-9505-ee4eaf1a5627-kube-api-access-55crl\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:29.454985 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:29.454981    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6578a06-3971-4811-9505-ee4eaf1a5627-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:29.768058 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:29.768027    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-278149b1-fk6vc"]
Nov 02 01:43:29.769231 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:43:29.769201    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd58675_c64e_4a5f_bb11_a87dd9f49852.slice/crio-d62eb857fa43171213c047f1374892dc2c92ce11429007354853062c63a7021e WatchSource:0}: Error finding container d62eb857fa43171213c047f1374892dc2c92ce11429007354853062c63a7021e: Status 404 returned error can't find the container with id d62eb857fa43171213c047f1374892dc2c92ce11429007354853062c63a7021e
Nov 02 01:43:30.117339 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:30.117300    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8408d89d-f6p4s" event={"ID":"c6578a06-3971-4811-9505-ee4eaf1a5627","Type":"ContainerDied","Data":"f2706eb90f6427816d69258aa46ad5a5e9ebcb98a1dc4617a5843974c8f00824"}
Nov 02 01:43:30.117790 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:30.117314    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8408d89d-f6p4s"
Nov 02 01:43:30.117790 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:30.117400    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2706eb90f6427816d69258aa46ad5a5e9ebcb98a1dc4617a5843974c8f00824"
Nov 02 01:43:30.119076 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:30.119049    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-278149b1-fk6vc" event={"ID":"8fd58675-c64e-4a5f-bb11-a87dd9f49852","Type":"ContainerStarted","Data":"32b11cf56ba7cd375c46a5733d83f33c48a1c9a778358c34006c5716cdba9328"}
Nov 02 01:43:30.119204 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:30.119102    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-278149b1-fk6vc" event={"ID":"8fd58675-c64e-4a5f-bb11-a87dd9f49852","Type":"ContainerStarted","Data":"d62eb857fa43171213c047f1374892dc2c92ce11429007354853062c63a7021e"}
Nov 02 01:43:30.236936 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:30.236887    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-278149b1-fk6vc" podStartSLOduration=1.236872893 podStartE2EDuration="1.236872893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:30.2341693 +0000 UTC m=+1075.137851427" watchObservedRunningTime="2026-11-02 01:43:30.236872893 +0000 UTC m=+1075.140555006"
Nov 02 01:43:31.127297 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:31.127248    5639 generic.go:356] "Generic (PLEG): container finished" podID="8fd58675-c64e-4a5f-bb11-a87dd9f49852" containerID="32b11cf56ba7cd375c46a5733d83f33c48a1c9a778358c34006c5716cdba9328" exitCode=0
Nov 02 01:43:31.127764 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:31.127343    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-278149b1-fk6vc" event={"ID":"8fd58675-c64e-4a5f-bb11-a87dd9f49852","Type":"ContainerDied","Data":"32b11cf56ba7cd375c46a5733d83f33c48a1c9a778358c34006c5716cdba9328"}
Nov 02 01:43:32.285170 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:32.285140    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-278149b1-fk6vc"
Nov 02 01:43:32.391714 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:32.391624    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fd58675-c64e-4a5f-bb11-a87dd9f49852-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fd58675-c64e-4a5f-bb11-a87dd9f49852-maas-api-openshift-ca\") pod \"8fd58675-c64e-4a5f-bb11-a87dd9f49852\" (UID: \"8fd58675-c64e-4a5f-bb11-a87dd9f49852\") "
Nov 02 01:43:32.391870 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:32.391820    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fd58675-c64e-4a5f-bb11-a87dd9f49852-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fd58675-c64e-4a5f-bb11-a87dd9f49852-maas-api-xks-ca\") pod \"8fd58675-c64e-4a5f-bb11-a87dd9f49852\" (UID: \"8fd58675-c64e-4a5f-bb11-a87dd9f49852\") "
Nov 02 01:43:32.391985 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:32.391969    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fd58675-c64e-4a5f-bb11-a87dd9f49852-kube-api-access-tkvql\" (UniqueName: \"kubernetes.io/projected/8fd58675-c64e-4a5f-bb11-a87dd9f49852-kube-api-access-tkvql\") pod \"8fd58675-c64e-4a5f-bb11-a87dd9f49852\" (UID: \"8fd58675-c64e-4a5f-bb11-a87dd9f49852\") "
Nov 02 01:43:32.392068 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:32.392039    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fd58675-c64e-4a5f-bb11-a87dd9f49852-maas-api-openshift-ca" pod "8fd58675-c64e-4a5f-bb11-a87dd9f49852" (UID: "8fd58675-c64e-4a5f-bb11-a87dd9f49852"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:32.392425 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:32.392401    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fd58675-c64e-4a5f-bb11-a87dd9f49852-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:32.394082 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:32.394060    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fd58675-c64e-4a5f-bb11-a87dd9f49852-maas-api-xks-ca" pod "8fd58675-c64e-4a5f-bb11-a87dd9f49852" (UID: "8fd58675-c64e-4a5f-bb11-a87dd9f49852"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:32.394200 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:32.394117    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fd58675-c64e-4a5f-bb11-a87dd9f49852-kube-api-access-tkvql" pod "8fd58675-c64e-4a5f-bb11-a87dd9f49852" (UID: "8fd58675-c64e-4a5f-bb11-a87dd9f49852"). InnerVolumeSpecName "kube-api-access-tkvql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:32.494238 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:32.494185    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fd58675-c64e-4a5f-bb11-a87dd9f49852-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:32.494238 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:32.494231    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkvql\" (UniqueName: \"kubernetes.io/projected/8fd58675-c64e-4a5f-bb11-a87dd9f49852-kube-api-access-tkvql\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:33.142008 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:33.141977    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-278149b1-fk6vc"
Nov 02 01:43:33.142008 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:33.141993    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-278149b1-fk6vc" event={"ID":"8fd58675-c64e-4a5f-bb11-a87dd9f49852","Type":"ContainerDied","Data":"d62eb857fa43171213c047f1374892dc2c92ce11429007354853062c63a7021e"}
Nov 02 01:43:33.142247 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:33.142041    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d62eb857fa43171213c047f1374892dc2c92ce11429007354853062c63a7021e"
Nov 02 01:43:38.579412 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.579374    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"]
Nov 02 01:43:38.580164 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.580147    5639 cpu_manager.go:414] "containerMap: removing container" podUID="c6578a06-3971-4811-9505-ee4eaf1a5627" containerName="revoke-keys"
Nov 02 01:43:38.580232 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.580167    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6578a06-3971-4811-9505-ee4eaf1a5627" containerName="revoke-keys"
Nov 02 01:43:38.580232 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.580183    5639 cpu_manager.go:414] "containerMap: removing container" podUID="8fd58675-c64e-4a5f-bb11-a87dd9f49852" containerName="revoke-keys"
Nov 02 01:43:38.580232 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.580190    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fd58675-c64e-4a5f-bb11-a87dd9f49852" containerName="revoke-keys"
Nov 02 01:43:38.580384 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.580370    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6578a06-3971-4811-9505-ee4eaf1a5627" containerName="revoke-keys"
Nov 02 01:43:38.580419 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.580388    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fd58675-c64e-4a5f-bb11-a87dd9f49852" containerName="revoke-keys"
Nov 02 01:43:38.585311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.585292    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.642177 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.642144    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-e06ea2-openshift-default-dockercfg-x8mvt\""
Nov 02 01:43:38.644296 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.644252    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"]
Nov 02 01:43:38.666630 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.666592    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.666784 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.666721    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.666835 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.666780    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.667074 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.666956    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.667074 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.666991    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945be3db-df2f-4ef9-9609-a7890bee84ce-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.667074 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.667030    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.667074 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.667057    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.667262 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.667091    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.667262 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.667212    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncx8f\" (UniqueName: \"kubernetes.io/projected/945be3db-df2f-4ef9-9609-a7890bee84ce-kube-api-access-ncx8f\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.768536 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.768500    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.768715 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.768554    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.768715 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.768595    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.768715 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.768668    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.768715 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.768684    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945be3db-df2f-4ef9-9609-a7890bee84ce-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.768715 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.768708    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.768927 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.768881    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.768982 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.768941    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.768982 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.768950    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.769081 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.768986    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.769081 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.769058    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncx8f\" (UniqueName: \"kubernetes.io/projected/945be3db-df2f-4ef9-9609-a7890bee84ce-kube-api-access-ncx8f\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.769164 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.769079    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.769329 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.769309    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.769513 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.769496    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945be3db-df2f-4ef9-9609-a7890bee84ce-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.771625 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.771599    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.771625 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.771615    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.830143 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.830067    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.830302 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.830230    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncx8f\" (UniqueName: \"kubernetes.io/projected/945be3db-df2f-4ef9-9609-a7890bee84ce-kube-api-access-ncx8f\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:38.899517 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:38.899480    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:39.243614 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.243534    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"]
Nov 02 01:43:39.249234 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.249207    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.309018 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.308968    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"]
Nov 02 01:43:39.341695 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.341663    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"]
Nov 02 01:43:39.341835 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:43:39.341664    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945be3db_df2f_4ef9_9609_a7890bee84ce.slice/crio-8d0808082e697ffbf18b3eff4fbc3e2aa1ffc29100914c5a4e9208c14ed57d50 WatchSource:0}: Error finding container 8d0808082e697ffbf18b3eff4fbc3e2aa1ffc29100914c5a4e9208c14ed57d50: Status 404 returned error can't find the container with id 8d0808082e697ffbf18b3eff4fbc3e2aa1ffc29100914c5a4e9208c14ed57d50
Nov 02 01:43:39.344563 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.344531    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:43:39.344672 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.344609    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:43:39.344672 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.344654    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:43:39.376377 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.376351    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istiod-ca-cert\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.376537 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.376469    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-data\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.376537 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.376504    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-workload-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.376692 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.376665    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcnrn\" (UniqueName: \"kubernetes.io/projected/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-kube-api-access-gcnrn\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.376812 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.376721    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-workload-certs\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.376812 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.376772    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-token\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.376938 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.376846    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-envoy\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.376938 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.376893    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-credential-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.377014 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.376940    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-podinfo\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.478735 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.478699    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-data\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.478915 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.478754    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-workload-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.478915 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.478842    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcnrn\" (UniqueName: \"kubernetes.io/projected/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-kube-api-access-gcnrn\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.478915 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.478892    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-workload-certs\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.479035 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.478926    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-token\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.479035 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.478968    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-envoy\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.479035 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.479004    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-credential-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.479186 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.479046    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-podinfo\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.479186 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.479068    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istiod-ca-cert\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.479894 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.479867    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istiod-ca-cert\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.480162 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.480142    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-data\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.480400 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.480381    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-workload-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.480914 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.480894    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-workload-certs\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.481660 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.481631    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-credential-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.483493 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.483466    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-envoy\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.483876 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.483854    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-podinfo\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.541233 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.541153    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcnrn\" (UniqueName: \"kubernetes.io/projected/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-kube-api-access-gcnrn\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.541859 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.541839    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-token\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:39.564142 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:39.564112    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:40.013265 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:40.001404    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"]
Nov 02 01:43:40.013265 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:43:40.002304    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c64df3_8d18_42b1_ba44_a8f0bc5ae652.slice/crio-6093efd3e55a8d2f56c0304bc87ea03064d7276b46b6e6be5c04d525c5a03a6e WatchSource:0}: Error finding container 6093efd3e55a8d2f56c0304bc87ea03064d7276b46b6e6be5c04d525c5a03a6e: Status 404 returned error can't find the container with id 6093efd3e55a8d2f56c0304bc87ea03064d7276b46b6e6be5c04d525c5a03a6e
Nov 02 01:43:40.013265 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:40.004855    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:43:40.013265 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:40.004918    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:43:40.013265 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:40.004944    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:43:40.198374 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:40.198334    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc" event={"ID":"08c64df3-8d18-42b1-ba44-a8f0bc5ae652","Type":"ContainerStarted","Data":"496173330f9a8113138bcce1342a857bbb3086f1e4b5fd18d8aab537e335bf6e"}
Nov 02 01:43:40.198574 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:40.198389    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc" event={"ID":"08c64df3-8d18-42b1-ba44-a8f0bc5ae652","Type":"ContainerStarted","Data":"6093efd3e55a8d2f56c0304bc87ea03064d7276b46b6e6be5c04d525c5a03a6e"}
Nov 02 01:43:40.200119 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:40.200097    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp" event={"ID":"945be3db-df2f-4ef9-9609-a7890bee84ce","Type":"ContainerStarted","Data":"e07eb0ed26cff5e32d54db412e9bea74ff8f87d80a1aceed083e11651b35168e"}
Nov 02 01:43:40.200190 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:40.200138    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp" event={"ID":"945be3db-df2f-4ef9-9609-a7890bee84ce","Type":"ContainerStarted","Data":"8d0808082e697ffbf18b3eff4fbc3e2aa1ffc29100914c5a4e9208c14ed57d50"}
Nov 02 01:43:40.323426 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:40.323308    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc" podStartSLOduration=1.32327342 podStartE2EDuration="1.32327342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:40.319254277 +0000 UTC m=+1085.222936393" watchObservedRunningTime="2026-11-02 01:43:40.32327342 +0000 UTC m=+1085.226955534"
Nov 02 01:43:40.445751 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:40.444597    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp" podStartSLOduration=2.444576838 podStartE2EDuration="2.444576838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:40.442565065 +0000 UTC m=+1085.346247180" watchObservedRunningTime="2026-11-02 01:43:40.444576838 +0000 UTC m=+1085.348258952"
Nov 02 01:43:40.568449 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:40.568412    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:40.909306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:40.909028    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:40.925813 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:40.925769    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 02 01:43:40.926125 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:40.926091    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp" podUID="945be3db-df2f-4ef9-9609-a7890bee84ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 02 01:43:41.578318 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:41.574254    5639 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 01:43:41.578318 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:41.574375    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc" podUID="08c64df3-8d18-42b1-ba44-a8f0bc5ae652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 01:43:41.857931 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:41.854813    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a4134a0-s666n"]
Nov 02 01:43:41.870969 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:41.868564    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4134a0-s666n"
Nov 02 01:43:41.903732 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:41.903318    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 02 01:43:41.903732 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:41.903383    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp" podUID="945be3db-df2f-4ef9-9609-a7890bee84ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 02 01:43:41.918416 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:41.918373    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a4134a0-s666n"]
Nov 02 01:43:41.939397 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:41.939364    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:43:41.953648 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:41.953615    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:43:41.997243 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:41.960732    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:43:41.997521 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:41.961439    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:43:42.107711 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:42.107674    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqt2s\" (UniqueName: \"kubernetes.io/projected/105ca4e7-8651-4327-98e1-34a27970d9b2-kube-api-access-kqt2s\") pod \"maas-api-revoke-sub-7a4134a0-s666n\" (UID: \"105ca4e7-8651-4327-98e1-34a27970d9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4134a0-s666n"
Nov 02 01:43:42.108816 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:42.108742    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/105ca4e7-8651-4327-98e1-34a27970d9b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a4134a0-s666n\" (UID: \"105ca4e7-8651-4327-98e1-34a27970d9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4134a0-s666n"
Nov 02 01:43:42.109056 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:42.109035    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/105ca4e7-8651-4327-98e1-34a27970d9b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a4134a0-s666n\" (UID: \"105ca4e7-8651-4327-98e1-34a27970d9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4134a0-s666n"
Nov 02 01:43:42.222986 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:42.222934    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqt2s\" (UniqueName: \"kubernetes.io/projected/105ca4e7-8651-4327-98e1-34a27970d9b2-kube-api-access-kqt2s\") pod \"maas-api-revoke-sub-7a4134a0-s666n\" (UID: \"105ca4e7-8651-4327-98e1-34a27970d9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4134a0-s666n"
Nov 02 01:43:42.240304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:42.238378    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/105ca4e7-8651-4327-98e1-34a27970d9b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a4134a0-s666n\" (UID: \"105ca4e7-8651-4327-98e1-34a27970d9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4134a0-s666n"
Nov 02 01:43:42.240304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:42.238431    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/105ca4e7-8651-4327-98e1-34a27970d9b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a4134a0-s666n\" (UID: \"105ca4e7-8651-4327-98e1-34a27970d9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4134a0-s666n"
Nov 02 01:43:42.240304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:42.239452    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/105ca4e7-8651-4327-98e1-34a27970d9b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a4134a0-s666n\" (UID: \"105ca4e7-8651-4327-98e1-34a27970d9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4134a0-s666n"
Nov 02 01:43:42.250399 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:42.250255    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/105ca4e7-8651-4327-98e1-34a27970d9b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a4134a0-s666n\" (UID: \"105ca4e7-8651-4327-98e1-34a27970d9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4134a0-s666n"
Nov 02 01:43:42.314360 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:42.313928    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqt2s\" (UniqueName: \"kubernetes.io/projected/105ca4e7-8651-4327-98e1-34a27970d9b2-kube-api-access-kqt2s\") pod \"maas-api-revoke-sub-7a4134a0-s666n\" (UID: \"105ca4e7-8651-4327-98e1-34a27970d9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4134a0-s666n"
Nov 02 01:43:42.527384 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:42.524502    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4134a0-s666n"
Nov 02 01:43:42.581661 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:42.581624    5639 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 01:43:42.582253 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:42.582221    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc" podUID="08c64df3-8d18-42b1-ba44-a8f0bc5ae652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 01:43:42.901916 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:42.901882    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 02 01:43:42.902238 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:42.902206    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp" podUID="945be3db-df2f-4ef9-9609-a7890bee84ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 02 01:43:43.336854 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:43.336810    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a4134a0-s666n"]
Nov 02 01:43:43.346233 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:43.346209    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4134a0-s666n" event={"ID":"105ca4e7-8651-4327-98e1-34a27970d9b2","Type":"ContainerStarted","Data":"6a06b39563d712596fc05023ca211b0291ae213dfb96e43776c752d499bf6442"}
Nov 02 01:43:43.568306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:43.567804    5639 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 01:43:43.568306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:43.567882    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc" podUID="08c64df3-8d18-42b1-ba44-a8f0bc5ae652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 01:43:43.904369 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:43.904051    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 02 01:43:43.904369 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:43.904114    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp" podUID="945be3db-df2f-4ef9-9609-a7890bee84ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 02 01:43:44.354616 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:44.354564    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4134a0-s666n" event={"ID":"105ca4e7-8651-4327-98e1-34a27970d9b2","Type":"ContainerStarted","Data":"bd7834aab877d34ff0a9911a7523a202f152fca18427f8e4fad545318cc964dd"}
Nov 02 01:43:44.503351 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:44.502490    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4134a0-s666n" podStartSLOduration=3.5024682179999997 podStartE2EDuration="3.502468218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:44.48201583 +0000 UTC m=+1089.385697944" watchObservedRunningTime="2026-11-02 01:43:44.502468218 +0000 UTC m=+1089.406150334"
Nov 02 01:43:44.573305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:44.569223    5639 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 01:43:44.573305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:44.569312    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc" podUID="08c64df3-8d18-42b1-ba44-a8f0bc5ae652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 01:43:44.900209 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:44.900175    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 02 01:43:44.900544 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:44.900500    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp" podUID="945be3db-df2f-4ef9-9609-a7890bee84ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 02 01:43:45.412358 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:45.410989    5639 generic.go:356] "Generic (PLEG): container finished" podID="105ca4e7-8651-4327-98e1-34a27970d9b2" containerID="bd7834aab877d34ff0a9911a7523a202f152fca18427f8e4fad545318cc964dd" exitCode=0
Nov 02 01:43:45.412358 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:45.411223    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4134a0-s666n" event={"ID":"105ca4e7-8651-4327-98e1-34a27970d9b2","Type":"ContainerDied","Data":"bd7834aab877d34ff0a9911a7523a202f152fca18427f8e4fad545318cc964dd"}
Nov 02 01:43:45.568728 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:45.568692    5639 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 01:43:45.569021 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:45.568992    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc" podUID="08c64df3-8d18-42b1-ba44-a8f0bc5ae652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 01:43:45.899992 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:45.899959    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 02 01:43:45.900179 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:45.900011    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp" podUID="945be3db-df2f-4ef9-9609-a7890bee84ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 02 01:43:46.565568 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:46.565048    5639 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 01:43:46.565568 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:46.565125    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc" podUID="08c64df3-8d18-42b1-ba44-a8f0bc5ae652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 01:43:46.597406 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:46.597374    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4134a0-s666n"
Nov 02 01:43:46.650319 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:46.649390    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/105ca4e7-8651-4327-98e1-34a27970d9b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/105ca4e7-8651-4327-98e1-34a27970d9b2-maas-api-xks-ca\") pod \"105ca4e7-8651-4327-98e1-34a27970d9b2\" (UID: \"105ca4e7-8651-4327-98e1-34a27970d9b2\") "
Nov 02 01:43:46.650319 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:46.649484    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/105ca4e7-8651-4327-98e1-34a27970d9b2-kube-api-access-kqt2s\" (UniqueName: \"kubernetes.io/projected/105ca4e7-8651-4327-98e1-34a27970d9b2-kube-api-access-kqt2s\") pod \"105ca4e7-8651-4327-98e1-34a27970d9b2\" (UID: \"105ca4e7-8651-4327-98e1-34a27970d9b2\") "
Nov 02 01:43:46.650319 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:46.649566    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/105ca4e7-8651-4327-98e1-34a27970d9b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/105ca4e7-8651-4327-98e1-34a27970d9b2-maas-api-openshift-ca\") pod \"105ca4e7-8651-4327-98e1-34a27970d9b2\" (UID: \"105ca4e7-8651-4327-98e1-34a27970d9b2\") "
Nov 02 01:43:46.650617 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:46.650359    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/105ca4e7-8651-4327-98e1-34a27970d9b2-maas-api-openshift-ca" pod "105ca4e7-8651-4327-98e1-34a27970d9b2" (UID: "105ca4e7-8651-4327-98e1-34a27970d9b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:46.662329 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:46.652480    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/105ca4e7-8651-4327-98e1-34a27970d9b2-maas-api-xks-ca" pod "105ca4e7-8651-4327-98e1-34a27970d9b2" (UID: "105ca4e7-8651-4327-98e1-34a27970d9b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:46.662329 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:46.652644    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105ca4e7-8651-4327-98e1-34a27970d9b2-kube-api-access-kqt2s" pod "105ca4e7-8651-4327-98e1-34a27970d9b2" (UID: "105ca4e7-8651-4327-98e1-34a27970d9b2"). InnerVolumeSpecName "kube-api-access-kqt2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:46.751950 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:46.751841    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/105ca4e7-8651-4327-98e1-34a27970d9b2-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:46.751950 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:46.751893    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqt2s\" (UniqueName: \"kubernetes.io/projected/105ca4e7-8651-4327-98e1-34a27970d9b2-kube-api-access-kqt2s\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:46.751950 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:46.751911    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/105ca4e7-8651-4327-98e1-34a27970d9b2-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:47.456554 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:47.453640    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4134a0-s666n" event={"ID":"105ca4e7-8651-4327-98e1-34a27970d9b2","Type":"ContainerDied","Data":"6a06b39563d712596fc05023ca211b0291ae213dfb96e43776c752d499bf6442"}
Nov 02 01:43:47.456554 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:47.453741    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a06b39563d712596fc05023ca211b0291ae213dfb96e43776c752d499bf6442"
Nov 02 01:43:47.456554 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:47.453861    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4134a0-s666n"
Nov 02 01:43:47.463802 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:47.462574    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:47.466318 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:47.464192    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:47.468756 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:47.468731    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:43:47.879763 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:47.879726    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:47.880429 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:47.880213    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:47.881773 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:43:47.881738    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:43:57.421995 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:43:57.421953    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:44:01.738762 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:01.738724    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:44:23.661636 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.656080    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"]
Nov 02 01:44:23.661636 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.656881    5639 cpu_manager.go:414] "containerMap: removing container" podUID="105ca4e7-8651-4327-98e1-34a27970d9b2" containerName="revoke-keys"
Nov 02 01:44:23.661636 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.656899    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="105ca4e7-8651-4327-98e1-34a27970d9b2" containerName="revoke-keys"
Nov 02 01:44:23.661636 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.657051    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="105ca4e7-8651-4327-98e1-34a27970d9b2" containerName="revoke-keys"
Nov 02 01:44:23.662849 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.662823    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.732503 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.732472    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.732664 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.732540    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.732664 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.732631    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.732736 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.732662    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw42f\" (UniqueName: \"kubernetes.io/projected/02d298db-5bc9-469a-af6b-ba7fa45f8602-kube-api-access-pw42f\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.732784 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.732767    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.732819 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.732800    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.732855 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.732816    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.732917 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.732900    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d298db-5bc9-469a-af6b-ba7fa45f8602-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.732951 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.732937    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.744481 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.744455    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-5888dc-openshift-default-dockercfg-cgmjh\""
Nov 02 01:44:23.768776 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.768737    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"]
Nov 02 01:44:23.833768 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.833737    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.833951 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.833785    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.833951 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.833804    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw42f\" (UniqueName: \"kubernetes.io/projected/02d298db-5bc9-469a-af6b-ba7fa45f8602-kube-api-access-pw42f\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.833951 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.833852    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.833951 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.833872    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.833951 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.833887    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.834236 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.833964    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d298db-5bc9-469a-af6b-ba7fa45f8602-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.834236 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.834001    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.834236 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.834120    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.834236 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.834205    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.834505 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.834309    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.834505 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.834387    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.834640 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.834571    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.834723 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.834702    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d298db-5bc9-469a-af6b-ba7fa45f8602-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.836669 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.836650    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.836783 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.836692    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.895864 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.895834    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw42f\" (UniqueName: \"kubernetes.io/projected/02d298db-5bc9-469a-af6b-ba7fa45f8602-kube-api-access-pw42f\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.896015 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.895839    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.934212 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.934140    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"]
Nov 02 01:44:23.939510 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.939489    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:23.975852 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.975826    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:23.998892 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:23.998863    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"]
Nov 02 01:44:24.040323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.038132    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.040323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.038201    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.040323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.038229    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22wr6\" (UniqueName: \"kubernetes.io/projected/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-kube-api-access-22wr6\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.040323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.038267    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.040323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.038453    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.040323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.038483    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.040323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.038591    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.040323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.038621    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.040323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.038696    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.139608 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.139574    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.139608 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.139611    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22wr6\" (UniqueName: \"kubernetes.io/projected/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-kube-api-access-22wr6\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.139851 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.139770    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.139851 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.139807    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.139851 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.139831    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.140002 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.139940    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.140002 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.139970    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.140099 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.140002    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.140099 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.140083    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.140233 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.140125    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.140306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.140240    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.140487 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.140453    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.140593 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.140466    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.140593 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.140531    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.142193 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.142172    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.142639 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.142622    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.204207 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.204132    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22wr6\" (UniqueName: \"kubernetes.io/projected/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-kube-api-access-22wr6\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.204841 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.204816    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.254545 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.254517    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:24.416181 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.416154    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"]
Nov 02 01:44:24.417779 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:44:24.417754    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02d298db_5bc9_469a_af6b_ba7fa45f8602.slice/crio-31bc616ea18fa5033ff6b6487ebee038ad63b4ab326586722ebe34e1320bafcf WatchSource:0}: Error finding container 31bc616ea18fa5033ff6b6487ebee038ad63b4ab326586722ebe34e1320bafcf: Status 404 returned error can't find the container with id 31bc616ea18fa5033ff6b6487ebee038ad63b4ab326586722ebe34e1320bafcf
Nov 02 01:44:24.420420 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.420381    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:44:24.420510 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.420450    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:44:24.420510 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.420491    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:44:24.644112 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:44:24.644084    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1640ee78_dc59_4afb_a7f6_e130f60cdf4a.slice/crio-b8b5b5e5fb58f22f5f63b708f3448eb10bf673334181aceaa11ba13882bf72a0 WatchSource:0}: Error finding container b8b5b5e5fb58f22f5f63b708f3448eb10bf673334181aceaa11ba13882bf72a0: Status 404 returned error can't find the container with id b8b5b5e5fb58f22f5f63b708f3448eb10bf673334181aceaa11ba13882bf72a0
Nov 02 01:44:24.644426 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.644401    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"]
Nov 02 01:44:24.647634 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.647594    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:44:24.647727 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.647686    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:44:24.647779 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.647763    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:44:24.717047 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.717000    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp" event={"ID":"02d298db-5bc9-469a-af6b-ba7fa45f8602","Type":"ContainerStarted","Data":"3d0d09d742168662cec0cff79ef1c1312b95607a52508c79d5b443c99cf1110d"}
Nov 02 01:44:24.717493 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.717070    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp" event={"ID":"02d298db-5bc9-469a-af6b-ba7fa45f8602","Type":"ContainerStarted","Data":"31bc616ea18fa5033ff6b6487ebee038ad63b4ab326586722ebe34e1320bafcf"}
Nov 02 01:44:24.718463 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.718441    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h" event={"ID":"1640ee78-dc59-4afb-a7f6-e130f60cdf4a","Type":"ContainerStarted","Data":"b8b5b5e5fb58f22f5f63b708f3448eb10bf673334181aceaa11ba13882bf72a0"}
Nov 02 01:44:24.842190 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.842125    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp" podStartSLOduration=1.842106282 podStartE2EDuration="1.842106282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:24.836403834 +0000 UTC m=+1129.740085951" watchObservedRunningTime="2026-11-02 01:44:24.842106282 +0000 UTC m=+1129.745788397"
Nov 02 01:44:24.976973 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:24.976885    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:25.732400 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:25.731650    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h" event={"ID":"1640ee78-dc59-4afb-a7f6-e130f60cdf4a","Type":"ContainerStarted","Data":"9ac49c154511182455be13f90b691f0238fa39d2a62e516a02dd8e425f2e0a64"}
Nov 02 01:44:25.903303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:25.899781    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h" podStartSLOduration=2.899759584 podStartE2EDuration="2.899759584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:25.869807405 +0000 UTC m=+1130.773489530" watchObservedRunningTime="2026-11-02 01:44:25.899759584 +0000 UTC m=+1130.803441697"
Nov 02 01:44:25.999492 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:25.999390    5639 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 02 01:44:25.999728 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:25.999702    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp" podUID="02d298db-5bc9-469a-af6b-ba7fa45f8602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 02 01:44:26.255816 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:26.255739    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:26.257152 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:26.257130    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 01:44:26.257217 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:26.257189    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h" podUID="1640ee78-dc59-4afb-a7f6-e130f60cdf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 01:44:26.980751 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:26.980712    5639 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 02 01:44:26.981380 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:26.981347    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp" podUID="02d298db-5bc9-469a-af6b-ba7fa45f8602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 02 01:44:27.256442 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:27.255688    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 01:44:27.256442 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:27.255752    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h" podUID="1640ee78-dc59-4afb-a7f6-e130f60cdf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 01:44:27.440214 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:44:27.440176    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:44:27.978730 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:27.978692    5639 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 02 01:44:27.979023 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:27.978996    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp" podUID="02d298db-5bc9-469a-af6b-ba7fa45f8602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 02 01:44:28.255937 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:28.255854    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 01:44:28.256497 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:28.256471    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h" podUID="1640ee78-dc59-4afb-a7f6-e130f60cdf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 01:44:28.384306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:28.381648    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:44:28.984687 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:28.982862    5639 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 02 01:44:28.984687 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:28.982936    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp" podUID="02d298db-5bc9-469a-af6b-ba7fa45f8602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 02 01:44:29.257754 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:29.257667    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 01:44:29.258221 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:29.258061    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h" podUID="1640ee78-dc59-4afb-a7f6-e130f60cdf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 01:44:29.983727 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:29.983689    5639 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 02 01:44:29.984042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:29.984013    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp" podUID="02d298db-5bc9-469a-af6b-ba7fa45f8602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 02 01:44:30.255723 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:30.255641    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 01:44:30.255723 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:30.255709    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h" podUID="1640ee78-dc59-4afb-a7f6-e130f60cdf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 01:44:30.976712 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:30.976673    5639 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 02 01:44:30.977327 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:30.977272    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp" podUID="02d298db-5bc9-469a-af6b-ba7fa45f8602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 02 01:44:31.255208 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:31.255111    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 01:44:31.255208 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:31.255186    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h" podUID="1640ee78-dc59-4afb-a7f6-e130f60cdf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 01:44:31.980469 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:31.980438    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:31.980943 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:31.980919    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:31.981840 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:31.981818    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:44:32.259816 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:32.259742    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:32.260045 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:32.260026    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:32.260653 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:32.260630    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:44:56.140877 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:44:56.140840    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:44:57.434501 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:44:57.434458    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:45:00.171774 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.171741    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq"]
Nov 02 01:45:00.176196 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.176169    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw"]
Nov 02 01:45:00.176382 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.176319    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq"
Nov 02 01:45:00.180242 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.180221    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893065-xgfxp"]
Nov 02 01:45:00.180389 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.180364    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw"
Nov 02 01:45:00.184510 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.184491    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-xgfxp"
Nov 02 01:45:00.221584 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.221553    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf"]
Nov 02 01:45:00.225641 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.225615    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw"]
Nov 02 01:45:00.225831 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.225729    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf"
Nov 02 01:45:00.228550 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.228525    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:45:00.228715 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.228545    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:45:00.228715 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.228661    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:45:00.229686 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.229667    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw"
Nov 02 01:45:00.263814 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.263777    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb9bg\" (UniqueName: \"kubernetes.io/projected/6c86d89a-15d9-4aa0-86be-a24ad9ea29de-kube-api-access-fb9bg\") pod \"maas-api-key-cleanup-29893065-xgfxp\" (UID: \"6c86d89a-15d9-4aa0-86be-a24ad9ea29de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-xgfxp"
Nov 02 01:45:00.263981 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.263820    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmcqj\" (UniqueName: \"kubernetes.io/projected/a73a71e7-0590-4e7f-aa7a-059d7b53b75a-kube-api-access-lmcqj\") pod \"maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw\" (UID: \"a73a71e7-0590-4e7f-aa7a-059d7b53b75a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw"
Nov 02 01:45:00.263981 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.263962    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bbv4\" (UniqueName: \"kubernetes.io/projected/509e47f7-b40b-4e9e-98be-d13fe4cb99c8-kube-api-access-7bbv4\") pod \"maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf\" (UID: \"509e47f7-b40b-4e9e-98be-d13fe4cb99c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf"
Nov 02 01:45:00.264094 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.264081    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7950aea7-d901-478a-a27e-f5798c935c66-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw\" (UID: \"7950aea7-d901-478a-a27e-f5798c935c66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw"
Nov 02 01:45:00.264131 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.264104    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3de286-fa4a-4ff9-be08-163ffb17d034-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq\" (UID: \"bc3de286-fa4a-4ff9-be08-163ffb17d034\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq"
Nov 02 01:45:00.264164 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.264132    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a73a71e7-0590-4e7f-aa7a-059d7b53b75a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw\" (UID: \"a73a71e7-0590-4e7f-aa7a-059d7b53b75a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw"
Nov 02 01:45:00.264215 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.264199    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c86d89a-15d9-4aa0-86be-a24ad9ea29de-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893065-xgfxp\" (UID: \"6c86d89a-15d9-4aa0-86be-a24ad9ea29de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-xgfxp"
Nov 02 01:45:00.264250 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.264239    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl9ln\" (UniqueName: \"kubernetes.io/projected/bc3de286-fa4a-4ff9-be08-163ffb17d034-kube-api-access-cl9ln\") pod \"maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq\" (UID: \"bc3de286-fa4a-4ff9-be08-163ffb17d034\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq"
Nov 02 01:45:00.264333 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.264315    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/509e47f7-b40b-4e9e-98be-d13fe4cb99c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf\" (UID: \"509e47f7-b40b-4e9e-98be-d13fe4cb99c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf"
Nov 02 01:45:00.264389 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.264364    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gcwx\" (UniqueName: \"kubernetes.io/projected/7950aea7-d901-478a-a27e-f5798c935c66-kube-api-access-4gcwx\") pod \"maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw\" (UID: \"7950aea7-d901-478a-a27e-f5798c935c66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw"
Nov 02 01:45:00.274556 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.274520    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq"]
Nov 02 01:45:00.279389 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.279363    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw"]
Nov 02 01:45:00.284227 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.284188    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893065-xgfxp"]
Nov 02 01:45:00.288392 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.288360    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf"]
Nov 02 01:45:00.328175 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.328139    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw"]
Nov 02 01:45:00.365944 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.365903    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bbv4\" (UniqueName: \"kubernetes.io/projected/509e47f7-b40b-4e9e-98be-d13fe4cb99c8-kube-api-access-7bbv4\") pod \"maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf\" (UID: \"509e47f7-b40b-4e9e-98be-d13fe4cb99c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf"
Nov 02 01:45:00.366156 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.365994    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7950aea7-d901-478a-a27e-f5798c935c66-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw\" (UID: \"7950aea7-d901-478a-a27e-f5798c935c66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw"
Nov 02 01:45:00.366156 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.366013    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3de286-fa4a-4ff9-be08-163ffb17d034-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq\" (UID: \"bc3de286-fa4a-4ff9-be08-163ffb17d034\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq"
Nov 02 01:45:00.366156 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.366030    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a73a71e7-0590-4e7f-aa7a-059d7b53b75a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw\" (UID: \"a73a71e7-0590-4e7f-aa7a-059d7b53b75a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw"
Nov 02 01:45:00.366156 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.366064    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c86d89a-15d9-4aa0-86be-a24ad9ea29de-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893065-xgfxp\" (UID: \"6c86d89a-15d9-4aa0-86be-a24ad9ea29de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-xgfxp"
Nov 02 01:45:00.366600 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.366228    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl9ln\" (UniqueName: \"kubernetes.io/projected/bc3de286-fa4a-4ff9-be08-163ffb17d034-kube-api-access-cl9ln\") pod \"maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq\" (UID: \"bc3de286-fa4a-4ff9-be08-163ffb17d034\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq"
Nov 02 01:45:00.366930 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.366911    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7950aea7-d901-478a-a27e-f5798c935c66-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw\" (UID: \"7950aea7-d901-478a-a27e-f5798c935c66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw"
Nov 02 01:45:00.366990 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.366956    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c86d89a-15d9-4aa0-86be-a24ad9ea29de-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893065-xgfxp\" (UID: \"6c86d89a-15d9-4aa0-86be-a24ad9ea29de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-xgfxp"
Nov 02 01:45:00.366990 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.366972    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3de286-fa4a-4ff9-be08-163ffb17d034-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq\" (UID: \"bc3de286-fa4a-4ff9-be08-163ffb17d034\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq"
Nov 02 01:45:00.367057 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.366981    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a73a71e7-0590-4e7f-aa7a-059d7b53b75a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw\" (UID: \"a73a71e7-0590-4e7f-aa7a-059d7b53b75a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw"
Nov 02 01:45:00.367057 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.367033    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/509e47f7-b40b-4e9e-98be-d13fe4cb99c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf\" (UID: \"509e47f7-b40b-4e9e-98be-d13fe4cb99c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf"
Nov 02 01:45:00.367139 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.367122    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gcwx\" (UniqueName: \"kubernetes.io/projected/7950aea7-d901-478a-a27e-f5798c935c66-kube-api-access-4gcwx\") pod \"maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw\" (UID: \"7950aea7-d901-478a-a27e-f5798c935c66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw"
Nov 02 01:45:00.367204 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.367189    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb9bg\" (UniqueName: \"kubernetes.io/projected/6c86d89a-15d9-4aa0-86be-a24ad9ea29de-kube-api-access-fb9bg\") pod \"maas-api-key-cleanup-29893065-xgfxp\" (UID: \"6c86d89a-15d9-4aa0-86be-a24ad9ea29de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-xgfxp"
Nov 02 01:45:00.367239 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.367229    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmcqj\" (UniqueName: \"kubernetes.io/projected/a73a71e7-0590-4e7f-aa7a-059d7b53b75a-kube-api-access-lmcqj\") pod \"maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw\" (UID: \"a73a71e7-0590-4e7f-aa7a-059d7b53b75a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw"
Nov 02 01:45:00.367343 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.367325    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/509e47f7-b40b-4e9e-98be-d13fe4cb99c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf\" (UID: \"509e47f7-b40b-4e9e-98be-d13fe4cb99c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf"
Nov 02 01:45:00.427305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.427215    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gcwx\" (UniqueName: \"kubernetes.io/projected/7950aea7-d901-478a-a27e-f5798c935c66-kube-api-access-4gcwx\") pod \"maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw\" (UID: \"7950aea7-d901-478a-a27e-f5798c935c66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw"
Nov 02 01:45:00.427482 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.427369    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bbv4\" (UniqueName: \"kubernetes.io/projected/509e47f7-b40b-4e9e-98be-d13fe4cb99c8-kube-api-access-7bbv4\") pod \"maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf\" (UID: \"509e47f7-b40b-4e9e-98be-d13fe4cb99c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf"
Nov 02 01:45:00.427482 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.427384    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl9ln\" (UniqueName: \"kubernetes.io/projected/bc3de286-fa4a-4ff9-be08-163ffb17d034-kube-api-access-cl9ln\") pod \"maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq\" (UID: \"bc3de286-fa4a-4ff9-be08-163ffb17d034\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq"
Nov 02 01:45:00.428423 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.428395    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb9bg\" (UniqueName: \"kubernetes.io/projected/6c86d89a-15d9-4aa0-86be-a24ad9ea29de-kube-api-access-fb9bg\") pod \"maas-api-key-cleanup-29893065-xgfxp\" (UID: \"6c86d89a-15d9-4aa0-86be-a24ad9ea29de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-xgfxp"
Nov 02 01:45:00.428770 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.428717    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmcqj\" (UniqueName: \"kubernetes.io/projected/a73a71e7-0590-4e7f-aa7a-059d7b53b75a-kube-api-access-lmcqj\") pod \"maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw\" (UID: \"a73a71e7-0590-4e7f-aa7a-059d7b53b75a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw"
Nov 02 01:45:00.492161 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.492118    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq"
Nov 02 01:45:00.499802 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.499775    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw"
Nov 02 01:45:00.507443 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.507421    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-xgfxp"
Nov 02 01:45:00.550346 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.540783    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf"
Nov 02 01:45:00.550678 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.550645    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw"
Nov 02 01:45:00.885932 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.885901    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq"]
Nov 02 01:45:00.886568 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:45:00.886543    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3de286_fa4a_4ff9_be08_163ffb17d034.slice/crio-4958200e93ae16412845ef995faaa294561c782b21b17eeaef19364821888b92 WatchSource:0}: Error finding container 4958200e93ae16412845ef995faaa294561c782b21b17eeaef19364821888b92: Status 404 returned error can't find the container with id 4958200e93ae16412845ef995faaa294561c782b21b17eeaef19364821888b92
Nov 02 01:45:00.935853 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.935781    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw"]
Nov 02 01:45:00.937677 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:45:00.937643    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7950aea7_d901_478a_a27e_f5798c935c66.slice/crio-20e9b1262fdbea1d2f752a9e0797a04c4afff04fa04abb01c0aafcf1ad5e43dc WatchSource:0}: Error finding container 20e9b1262fdbea1d2f752a9e0797a04c4afff04fa04abb01c0aafcf1ad5e43dc: Status 404 returned error can't find the container with id 20e9b1262fdbea1d2f752a9e0797a04c4afff04fa04abb01c0aafcf1ad5e43dc
Nov 02 01:45:00.938878 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:45:00.938851    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c86d89a_15d9_4aa0_86be_a24ad9ea29de.slice/crio-223306c1e9b280850adf16d9a1b658e96e78e3b484b4dfaaec709d11e1187ccf WatchSource:0}: Error finding container 223306c1e9b280850adf16d9a1b658e96e78e3b484b4dfaaec709d11e1187ccf: Status 404 returned error can't find the container with id 223306c1e9b280850adf16d9a1b658e96e78e3b484b4dfaaec709d11e1187ccf
Nov 02 01:45:00.940084 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:00.940005    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893065-xgfxp"]
Nov 02 01:45:01.000610 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:01.000580    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf"]
Nov 02 01:45:01.018734 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:01.018702    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq" event={"ID":"bc3de286-fa4a-4ff9-be08-163ffb17d034","Type":"ContainerStarted","Data":"4958200e93ae16412845ef995faaa294561c782b21b17eeaef19364821888b92"}
Nov 02 01:45:01.019959 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:01.019937    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw" event={"ID":"7950aea7-d901-478a-a27e-f5798c935c66","Type":"ContainerStarted","Data":"20e9b1262fdbea1d2f752a9e0797a04c4afff04fa04abb01c0aafcf1ad5e43dc"}
Nov 02 01:45:01.020124 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:01.020106    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw"]
Nov 02 01:45:01.021233 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:01.021209    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-xgfxp" event={"ID":"6c86d89a-15d9-4aa0-86be-a24ad9ea29de","Type":"ContainerStarted","Data":"223306c1e9b280850adf16d9a1b658e96e78e3b484b4dfaaec709d11e1187ccf"}
Nov 02 01:45:01.062926 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:45:01.062886    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod509e47f7_b40b_4e9e_98be_d13fe4cb99c8.slice/crio-50997caed3e19f90409458b465e74034fe31f12f7f5cef364feb29121b87f164 WatchSource:0}: Error finding container 50997caed3e19f90409458b465e74034fe31f12f7f5cef364feb29121b87f164: Status 404 returned error can't find the container with id 50997caed3e19f90409458b465e74034fe31f12f7f5cef364feb29121b87f164
Nov 02 01:45:01.063423 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:45:01.063393    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda73a71e7_0590_4e7f_aa7a_059d7b53b75a.slice/crio-ff005c5432cac209ae615ba8c0c498915e2553ce52cc8b1a4d77e8f57ad43860 WatchSource:0}: Error finding container ff005c5432cac209ae615ba8c0c498915e2553ce52cc8b1a4d77e8f57ad43860: Status 404 returned error can't find the container with id ff005c5432cac209ae615ba8c0c498915e2553ce52cc8b1a4d77e8f57ad43860
Nov 02 01:45:02.029606 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:02.029568    5639 generic.go:356] "Generic (PLEG): container finished" podID="6c86d89a-15d9-4aa0-86be-a24ad9ea29de" containerID="c6c07e42f60f778e9f8627fbca98a7d4e77dc2fab0ff22fbdc61027282c34562" exitCode=60
Nov 02 01:45:02.030064 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:02.029664    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-xgfxp" event={"ID":"6c86d89a-15d9-4aa0-86be-a24ad9ea29de","Type":"ContainerDied","Data":"c6c07e42f60f778e9f8627fbca98a7d4e77dc2fab0ff22fbdc61027282c34562"}
Nov 02 01:45:02.030064 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:02.029942    5639 scope.go:122] "RemoveContainer" containerID="c6c07e42f60f778e9f8627fbca98a7d4e77dc2fab0ff22fbdc61027282c34562"
Nov 02 01:45:02.031675 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:02.031611    5639 generic.go:356] "Generic (PLEG): container finished" podID="bc3de286-fa4a-4ff9-be08-163ffb17d034" containerID="a709bf846a2f871d300bcafeacdf97f2fcd283719ca67d53b3899f45c733250f" exitCode=60
Nov 02 01:45:02.031738 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:02.031691    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq" event={"ID":"bc3de286-fa4a-4ff9-be08-163ffb17d034","Type":"ContainerDied","Data":"a709bf846a2f871d300bcafeacdf97f2fcd283719ca67d53b3899f45c733250f"}
Nov 02 01:45:02.031942 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:02.031926    5639 scope.go:122] "RemoveContainer" containerID="a709bf846a2f871d300bcafeacdf97f2fcd283719ca67d53b3899f45c733250f"
Nov 02 01:45:02.033441 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:02.033421    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf" event={"ID":"509e47f7-b40b-4e9e-98be-d13fe4cb99c8","Type":"ContainerStarted","Data":"d8727845e96e98c8f64619e15a27bba1b2d53cd115e386c1ebbf620a72eda23d"}
Nov 02 01:45:02.033497 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:02.033480    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf" event={"ID":"509e47f7-b40b-4e9e-98be-d13fe4cb99c8","Type":"ContainerStarted","Data":"50997caed3e19f90409458b465e74034fe31f12f7f5cef364feb29121b87f164"}
Nov 02 01:45:02.035216 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:02.035197    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw" event={"ID":"a73a71e7-0590-4e7f-aa7a-059d7b53b75a","Type":"ContainerStarted","Data":"2efdde73d561beacd538a363afeeba343b21666a766637b76d86369591cddf4f"}
Nov 02 01:45:02.035305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:02.035248    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw" event={"ID":"a73a71e7-0590-4e7f-aa7a-059d7b53b75a","Type":"ContainerStarted","Data":"ff005c5432cac209ae615ba8c0c498915e2553ce52cc8b1a4d77e8f57ad43860"}
Nov 02 01:45:02.036855 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:02.036835    5639 generic.go:356] "Generic (PLEG): container finished" podID="7950aea7-d901-478a-a27e-f5798c935c66" containerID="9e03390316a6670dc4841a77251cc5170e1d2812a909ff927ffb2aa6857297a4" exitCode=60
Nov 02 01:45:02.036935 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:02.036871    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw" event={"ID":"7950aea7-d901-478a-a27e-f5798c935c66","Type":"ContainerDied","Data":"9e03390316a6670dc4841a77251cc5170e1d2812a909ff927ffb2aa6857297a4"}
Nov 02 01:45:02.037058 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:02.037044    5639 scope.go:122] "RemoveContainer" containerID="9e03390316a6670dc4841a77251cc5170e1d2812a909ff927ffb2aa6857297a4"
Nov 02 01:45:02.264763 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:02.264702    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf" podStartSLOduration=2.264681903 podStartE2EDuration="2.264681903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:02.260769464 +0000 UTC m=+1167.164451590" watchObservedRunningTime="2026-11-02 01:45:02.264681903 +0000 UTC m=+1167.168364020"
Nov 02 01:45:02.379589 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:02.379527    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw" podStartSLOduration=2.379507859 podStartE2EDuration="2.379507859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:02.378246151 +0000 UTC m=+1167.281928264" watchObservedRunningTime="2026-11-02 01:45:02.379507859 +0000 UTC m=+1167.283189974"
Nov 02 01:45:03.049785 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:03.049749    5639 generic.go:356] "Generic (PLEG): container finished" podID="bc3de286-fa4a-4ff9-be08-163ffb17d034" containerID="ebd0a8011ce96fb217bb852810e6af5182c07e6aa5b92663019ebe7cb239e9f9" exitCode=60
Nov 02 01:45:03.050215 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:03.049830    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq" event={"ID":"bc3de286-fa4a-4ff9-be08-163ffb17d034","Type":"ContainerDied","Data":"ebd0a8011ce96fb217bb852810e6af5182c07e6aa5b92663019ebe7cb239e9f9"}
Nov 02 01:45:03.050215 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:03.049939    5639 scope.go:122] "RemoveContainer" containerID="a709bf846a2f871d300bcafeacdf97f2fcd283719ca67d53b3899f45c733250f"
Nov 02 01:45:03.050366 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:03.050328    5639 scope.go:122] "RemoveContainer" containerID="ebd0a8011ce96fb217bb852810e6af5182c07e6aa5b92663019ebe7cb239e9f9"
Nov 02 01:45:03.050712 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:45:03.050687    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq_odh-ai-gateway-infra(bc3de286-fa4a-4ff9-be08-163ffb17d034)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq" podUID="bc3de286-fa4a-4ff9-be08-163ffb17d034"
Nov 02 01:45:03.051682 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:03.051660    5639 generic.go:356] "Generic (PLEG): container finished" podID="509e47f7-b40b-4e9e-98be-d13fe4cb99c8" containerID="d8727845e96e98c8f64619e15a27bba1b2d53cd115e386c1ebbf620a72eda23d" exitCode=60
Nov 02 01:45:03.051787 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:03.051737    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf" event={"ID":"509e47f7-b40b-4e9e-98be-d13fe4cb99c8","Type":"ContainerDied","Data":"d8727845e96e98c8f64619e15a27bba1b2d53cd115e386c1ebbf620a72eda23d"}
Nov 02 01:45:03.051999 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:03.051983    5639 scope.go:122] "RemoveContainer" containerID="d8727845e96e98c8f64619e15a27bba1b2d53cd115e386c1ebbf620a72eda23d"
Nov 02 01:45:03.053390 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:03.053369    5639 generic.go:356] "Generic (PLEG): container finished" podID="a73a71e7-0590-4e7f-aa7a-059d7b53b75a" containerID="2efdde73d561beacd538a363afeeba343b21666a766637b76d86369591cddf4f" exitCode=60
Nov 02 01:45:03.053508 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:03.053465    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw" event={"ID":"a73a71e7-0590-4e7f-aa7a-059d7b53b75a","Type":"ContainerDied","Data":"2efdde73d561beacd538a363afeeba343b21666a766637b76d86369591cddf4f"}
Nov 02 01:45:03.053673 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:03.053659    5639 scope.go:122] "RemoveContainer" containerID="2efdde73d561beacd538a363afeeba343b21666a766637b76d86369591cddf4f"
Nov 02 01:45:03.055645 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:03.055618    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw" event={"ID":"7950aea7-d901-478a-a27e-f5798c935c66","Type":"ContainerStarted","Data":"28321a58eed9a62675e47da98ea74f56e96a45b710f220cee37f3ed3ac0a7d24"}
Nov 02 01:45:03.057658 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:03.057638    5639 generic.go:356] "Generic (PLEG): container finished" podID="6c86d89a-15d9-4aa0-86be-a24ad9ea29de" containerID="7bc1d5ba6bf0d5363b5a92e75ea1bb2e99830bf8f662a638b4ac717d477eda55" exitCode=60
Nov 02 01:45:03.057746 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:03.057720    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-xgfxp" event={"ID":"6c86d89a-15d9-4aa0-86be-a24ad9ea29de","Type":"ContainerDied","Data":"7bc1d5ba6bf0d5363b5a92e75ea1bb2e99830bf8f662a638b4ac717d477eda55"}
Nov 02 01:45:03.057949 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:03.057932    5639 scope.go:122] "RemoveContainer" containerID="7bc1d5ba6bf0d5363b5a92e75ea1bb2e99830bf8f662a638b4ac717d477eda55"
Nov 02 01:45:03.058141 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:45:03.058125    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893065-xgfxp_odh-ai-gateway-infra(6c86d89a-15d9-4aa0-86be-a24ad9ea29de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-xgfxp" podUID="6c86d89a-15d9-4aa0-86be-a24ad9ea29de"
Nov 02 01:45:03.068071 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:03.068047    5639 scope.go:122] "RemoveContainer" containerID="c6c07e42f60f778e9f8627fbca98a7d4e77dc2fab0ff22fbdc61027282c34562"
Nov 02 01:45:03.510529 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:03.510475    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw" podStartSLOduration=3.510459141 podStartE2EDuration="3.510459141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:03.508679627 +0000 UTC m=+1168.412361739" watchObservedRunningTime="2026-11-02 01:45:03.510459141 +0000 UTC m=+1168.414141324"
Nov 02 01:45:04.067381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:04.067355    5639 scope.go:122] "RemoveContainer" containerID="ebd0a8011ce96fb217bb852810e6af5182c07e6aa5b92663019ebe7cb239e9f9"
Nov 02 01:45:04.067797 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:45:04.067733    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq_odh-ai-gateway-infra(bc3de286-fa4a-4ff9-be08-163ffb17d034)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq" podUID="bc3de286-fa4a-4ff9-be08-163ffb17d034"
Nov 02 01:45:04.069198 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:04.069165    5639 generic.go:356] "Generic (PLEG): container finished" podID="509e47f7-b40b-4e9e-98be-d13fe4cb99c8" containerID="b7106fdb42836b475fbdf62c7e254da17f818ed438f9410d0125d52dc12e7feb" exitCode=60
Nov 02 01:45:04.069352 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:04.069246    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf" event={"ID":"509e47f7-b40b-4e9e-98be-d13fe4cb99c8","Type":"ContainerDied","Data":"b7106fdb42836b475fbdf62c7e254da17f818ed438f9410d0125d52dc12e7feb"}
Nov 02 01:45:04.069352 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:04.069345    5639 scope.go:122] "RemoveContainer" containerID="d8727845e96e98c8f64619e15a27bba1b2d53cd115e386c1ebbf620a72eda23d"
Nov 02 01:45:04.069623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:04.069605    5639 scope.go:122] "RemoveContainer" containerID="b7106fdb42836b475fbdf62c7e254da17f818ed438f9410d0125d52dc12e7feb"
Nov 02 01:45:04.069967 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:45:04.069944    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf_odh-ai-gateway-infra(509e47f7-b40b-4e9e-98be-d13fe4cb99c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf" podUID="509e47f7-b40b-4e9e-98be-d13fe4cb99c8"
Nov 02 01:45:04.071660 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:04.071633    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw" event={"ID":"a73a71e7-0590-4e7f-aa7a-059d7b53b75a","Type":"ContainerStarted","Data":"3a7bc99ac08bb7756e70a4b27d3649ce242c1a6b5134fd43b66db6688fd1b6b6"}
Nov 02 01:45:04.071884 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:04.071871    5639 scope.go:122] "RemoveContainer" containerID="3a7bc99ac08bb7756e70a4b27d3649ce242c1a6b5134fd43b66db6688fd1b6b6"
Nov 02 01:45:04.072145 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:45:04.072121    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw_odh-ai-gateway-infra(a73a71e7-0590-4e7f-aa7a-059d7b53b75a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw" podUID="a73a71e7-0590-4e7f-aa7a-059d7b53b75a"
Nov 02 01:45:04.073876 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:04.073809    5639 generic.go:356] "Generic (PLEG): container finished" podID="7950aea7-d901-478a-a27e-f5798c935c66" containerID="28321a58eed9a62675e47da98ea74f56e96a45b710f220cee37f3ed3ac0a7d24" exitCode=60
Nov 02 01:45:04.073952 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:04.073886    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw" event={"ID":"7950aea7-d901-478a-a27e-f5798c935c66","Type":"ContainerDied","Data":"28321a58eed9a62675e47da98ea74f56e96a45b710f220cee37f3ed3ac0a7d24"}
Nov 02 01:45:04.074387 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:04.074319    5639 scope.go:122] "RemoveContainer" containerID="28321a58eed9a62675e47da98ea74f56e96a45b710f220cee37f3ed3ac0a7d24"
Nov 02 01:45:04.074674 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:45:04.074646    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw_odh-ai-gateway-infra(7950aea7-d901-478a-a27e-f5798c935c66)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw" podUID="7950aea7-d901-478a-a27e-f5798c935c66"
Nov 02 01:45:04.075915 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:04.075900    5639 scope.go:122] "RemoveContainer" containerID="7bc1d5ba6bf0d5363b5a92e75ea1bb2e99830bf8f662a638b4ac717d477eda55"
Nov 02 01:45:04.076125 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:45:04.076110    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893065-xgfxp_odh-ai-gateway-infra(6c86d89a-15d9-4aa0-86be-a24ad9ea29de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-xgfxp" podUID="6c86d89a-15d9-4aa0-86be-a24ad9ea29de"
Nov 02 01:45:04.115959 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:04.115939    5639 scope.go:122] "RemoveContainer" containerID="9e03390316a6670dc4841a77251cc5170e1d2812a909ff927ffb2aa6857297a4"
Nov 02 01:45:05.086194 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:05.086157    5639 scope.go:122] "RemoveContainer" containerID="b7106fdb42836b475fbdf62c7e254da17f818ed438f9410d0125d52dc12e7feb"
Nov 02 01:45:05.086722 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:45:05.086544    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf_odh-ai-gateway-infra(509e47f7-b40b-4e9e-98be-d13fe4cb99c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf" podUID="509e47f7-b40b-4e9e-98be-d13fe4cb99c8"
Nov 02 01:45:05.087981 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:05.087951    5639 generic.go:356] "Generic (PLEG): container finished" podID="a73a71e7-0590-4e7f-aa7a-059d7b53b75a" containerID="3a7bc99ac08bb7756e70a4b27d3649ce242c1a6b5134fd43b66db6688fd1b6b6" exitCode=60
Nov 02 01:45:05.088132 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:05.088038    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw" event={"ID":"a73a71e7-0590-4e7f-aa7a-059d7b53b75a","Type":"ContainerDied","Data":"3a7bc99ac08bb7756e70a4b27d3649ce242c1a6b5134fd43b66db6688fd1b6b6"}
Nov 02 01:45:05.088132 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:05.088096    5639 scope.go:122] "RemoveContainer" containerID="2efdde73d561beacd538a363afeeba343b21666a766637b76d86369591cddf4f"
Nov 02 01:45:05.088315 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:05.088298    5639 scope.go:122] "RemoveContainer" containerID="3a7bc99ac08bb7756e70a4b27d3649ce242c1a6b5134fd43b66db6688fd1b6b6"
Nov 02 01:45:05.088632 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:45:05.088614    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw_odh-ai-gateway-infra(a73a71e7-0590-4e7f-aa7a-059d7b53b75a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw" podUID="a73a71e7-0590-4e7f-aa7a-059d7b53b75a"
Nov 02 01:45:05.090106 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:05.090090    5639 scope.go:122] "RemoveContainer" containerID="28321a58eed9a62675e47da98ea74f56e96a45b710f220cee37f3ed3ac0a7d24"
Nov 02 01:45:05.090371 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:45:05.090354    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw_odh-ai-gateway-infra(7950aea7-d901-478a-a27e-f5798c935c66)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw" podUID="7950aea7-d901-478a-a27e-f5798c935c66"
Nov 02 01:45:06.098918 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:06.098890    5639 scope.go:122] "RemoveContainer" containerID="3a7bc99ac08bb7756e70a4b27d3649ce242c1a6b5134fd43b66db6688fd1b6b6"
Nov 02 01:45:06.099323 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:45:06.099232    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw_odh-ai-gateway-infra(a73a71e7-0590-4e7f-aa7a-059d7b53b75a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw" podUID="a73a71e7-0590-4e7f-aa7a-059d7b53b75a"
Nov 02 01:45:14.211803 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:14.211769    5639 scope.go:122] "RemoveContainer" containerID="ebd0a8011ce96fb217bb852810e6af5182c07e6aa5b92663019ebe7cb239e9f9"
Nov 02 01:45:15.187780 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:15.187746    5639 generic.go:356] "Generic (PLEG): container finished" podID="bc3de286-fa4a-4ff9-be08-163ffb17d034" containerID="c5246c5e9cd3325bd22d5337670409799480603a0f5bb9f097176e53e05a697d" exitCode=60
Nov 02 01:45:15.187963 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:15.187822    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq" event={"ID":"bc3de286-fa4a-4ff9-be08-163ffb17d034","Type":"ContainerDied","Data":"c5246c5e9cd3325bd22d5337670409799480603a0f5bb9f097176e53e05a697d"}
Nov 02 01:45:15.187963 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:15.187882    5639 scope.go:122] "RemoveContainer" containerID="ebd0a8011ce96fb217bb852810e6af5182c07e6aa5b92663019ebe7cb239e9f9"
Nov 02 01:45:15.188377 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:15.188355    5639 scope.go:122] "RemoveContainer" containerID="c5246c5e9cd3325bd22d5337670409799480603a0f5bb9f097176e53e05a697d"
Nov 02 01:45:15.188724 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:45:15.188696    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq_odh-ai-gateway-infra(bc3de286-fa4a-4ff9-be08-163ffb17d034)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq" podUID="bc3de286-fa4a-4ff9-be08-163ffb17d034"
Nov 02 01:45:15.211267 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:15.211244    5639 scope.go:122] "RemoveContainer" containerID="28321a58eed9a62675e47da98ea74f56e96a45b710f220cee37f3ed3ac0a7d24"
Nov 02 01:45:15.211424 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:15.211405    5639 scope.go:122] "RemoveContainer" containerID="7bc1d5ba6bf0d5363b5a92e75ea1bb2e99830bf8f662a638b4ac717d477eda55"
Nov 02 01:45:16.198247 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:16.198209    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw" event={"ID":"7950aea7-d901-478a-a27e-f5798c935c66","Type":"ContainerStarted","Data":"c1481e673299d9754f204395435f89d285fb39a33002d35e2a16a97f98a58786"}
Nov 02 01:45:16.200239 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:16.200212    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-xgfxp" event={"ID":"6c86d89a-15d9-4aa0-86be-a24ad9ea29de","Type":"ContainerStarted","Data":"0edf4887103f4f4fc3479b8eaee07fe1b95fbf8c320df3f2db7f5ef4e26caa5b"}
Nov 02 01:45:16.317089 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:16.317049    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq"]
Nov 02 01:45:16.489541 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:16.489516    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq"
Nov 02 01:45:16.592935 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:16.592899    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc3de286-fa4a-4ff9-be08-163ffb17d034-kube-api-access-cl9ln\" (UniqueName: \"kubernetes.io/projected/bc3de286-fa4a-4ff9-be08-163ffb17d034-kube-api-access-cl9ln\") pod \"bc3de286-fa4a-4ff9-be08-163ffb17d034\" (UID: \"bc3de286-fa4a-4ff9-be08-163ffb17d034\") "
Nov 02 01:45:16.593107 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:16.593022    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc3de286-fa4a-4ff9-be08-163ffb17d034-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3de286-fa4a-4ff9-be08-163ffb17d034-maas-api-service-ca\") pod \"bc3de286-fa4a-4ff9-be08-163ffb17d034\" (UID: \"bc3de286-fa4a-4ff9-be08-163ffb17d034\") "
Nov 02 01:45:16.593421 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:16.593395    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc3de286-fa4a-4ff9-be08-163ffb17d034-maas-api-service-ca" pod "bc3de286-fa4a-4ff9-be08-163ffb17d034" (UID: "bc3de286-fa4a-4ff9-be08-163ffb17d034"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:16.593645 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:16.593627    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3de286-fa4a-4ff9-be08-163ffb17d034-maas-api-service-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:45:16.595779 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:16.595750    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc3de286-fa4a-4ff9-be08-163ffb17d034-kube-api-access-cl9ln" pod "bc3de286-fa4a-4ff9-be08-163ffb17d034" (UID: "bc3de286-fa4a-4ff9-be08-163ffb17d034"). InnerVolumeSpecName "kube-api-access-cl9ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:16.596548 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:16.596506    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-xgfxp" podStartSLOduration=16.596492995 podStartE2EDuration="16.596492995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:16.595339248 +0000 UTC m=+1181.499021364" watchObservedRunningTime="2026-11-02 01:45:16.596492995 +0000 UTC m=+1181.500175122"
Nov 02 01:45:16.695324 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:16.695246    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl9ln\" (UniqueName: \"kubernetes.io/projected/bc3de286-fa4a-4ff9-be08-163ffb17d034-kube-api-access-cl9ln\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:45:17.209743 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:17.209702    5639 generic.go:356] "Generic (PLEG): container finished" podID="7950aea7-d901-478a-a27e-f5798c935c66" containerID="c1481e673299d9754f204395435f89d285fb39a33002d35e2a16a97f98a58786" exitCode=60
Nov 02 01:45:17.210243 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:17.209777    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw" event={"ID":"7950aea7-d901-478a-a27e-f5798c935c66","Type":"ContainerDied","Data":"c1481e673299d9754f204395435f89d285fb39a33002d35e2a16a97f98a58786"}
Nov 02 01:45:17.210243 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:17.209879    5639 scope.go:122] "RemoveContainer" containerID="28321a58eed9a62675e47da98ea74f56e96a45b710f220cee37f3ed3ac0a7d24"
Nov 02 01:45:17.210399 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:17.210262    5639 scope.go:122] "RemoveContainer" containerID="c1481e673299d9754f204395435f89d285fb39a33002d35e2a16a97f98a58786"
Nov 02 01:45:17.210697 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:45:17.210662    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw_odh-ai-gateway-infra(7950aea7-d901-478a-a27e-f5798c935c66)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw" podUID="7950aea7-d901-478a-a27e-f5798c935c66"
Nov 02 01:45:17.212161 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:17.211618    5639 scope.go:122] "RemoveContainer" containerID="b7106fdb42836b475fbdf62c7e254da17f818ed438f9410d0125d52dc12e7feb"
Nov 02 01:45:17.212161 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:17.212082    5639 generic.go:356] "Generic (PLEG): container finished" podID="6c86d89a-15d9-4aa0-86be-a24ad9ea29de" containerID="0edf4887103f4f4fc3479b8eaee07fe1b95fbf8c320df3f2db7f5ef4e26caa5b" exitCode=60
Nov 02 01:45:17.214375 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:17.214337    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq"
Nov 02 01:45:17.222633 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:17.222603    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-xgfxp" event={"ID":"6c86d89a-15d9-4aa0-86be-a24ad9ea29de","Type":"ContainerDied","Data":"0edf4887103f4f4fc3479b8eaee07fe1b95fbf8c320df3f2db7f5ef4e26caa5b"}
Nov 02 01:45:17.222769 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:17.222667    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq" event={"ID":"bc3de286-fa4a-4ff9-be08-163ffb17d034","Type":"ContainerDied","Data":"4958200e93ae16412845ef995faaa294561c782b21b17eeaef19364821888b92"}
Nov 02 01:45:17.223042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:17.223027    5639 scope.go:122] "RemoveContainer" containerID="0edf4887103f4f4fc3479b8eaee07fe1b95fbf8c320df3f2db7f5ef4e26caa5b"
Nov 02 01:45:17.223336 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:45:17.223316    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893065-xgfxp_odh-ai-gateway-infra(6c86d89a-15d9-4aa0-86be-a24ad9ea29de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-xgfxp" podUID="6c86d89a-15d9-4aa0-86be-a24ad9ea29de"
Nov 02 01:45:17.228065 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:17.228048    5639 scope.go:122] "RemoveContainer" containerID="7bc1d5ba6bf0d5363b5a92e75ea1bb2e99830bf8f662a638b4ac717d477eda55"
Nov 02 01:45:17.250149 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:17.250127    5639 scope.go:122] "RemoveContainer" containerID="c5246c5e9cd3325bd22d5337670409799480603a0f5bb9f097176e53e05a697d"
Nov 02 01:45:17.329114 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:17.329079    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw"]
Nov 02 01:45:17.453749 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:17.453706    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq"]
Nov 02 01:45:17.457510 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:17.457484    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893065-9lmxq"]
Nov 02 01:45:17.605804 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:17.605761    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893065-xgfxp"]
Nov 02 01:45:18.211594 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:18.211554    5639 scope.go:122] "RemoveContainer" containerID="3a7bc99ac08bb7756e70a4b27d3649ce242c1a6b5134fd43b66db6688fd1b6b6"
Nov 02 01:45:18.235625 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:18.235592    5639 generic.go:356] "Generic (PLEG): container finished" podID="509e47f7-b40b-4e9e-98be-d13fe4cb99c8" containerID="bc7267a6398a8fce28f06fe779c2c23447aa345bf7352183dc91835d22d062c7" exitCode=60
Nov 02 01:45:18.235786 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:18.235660    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf" event={"ID":"509e47f7-b40b-4e9e-98be-d13fe4cb99c8","Type":"ContainerDied","Data":"bc7267a6398a8fce28f06fe779c2c23447aa345bf7352183dc91835d22d062c7"}
Nov 02 01:45:18.235786 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:18.235715    5639 scope.go:122] "RemoveContainer" containerID="b7106fdb42836b475fbdf62c7e254da17f818ed438f9410d0125d52dc12e7feb"
Nov 02 01:45:18.236202 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:18.236180    5639 scope.go:122] "RemoveContainer" containerID="bc7267a6398a8fce28f06fe779c2c23447aa345bf7352183dc91835d22d062c7"
Nov 02 01:45:18.236626 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:45:18.236598    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf_odh-ai-gateway-infra(509e47f7-b40b-4e9e-98be-d13fe4cb99c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf" podUID="509e47f7-b40b-4e9e-98be-d13fe4cb99c8"
Nov 02 01:45:18.413592 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:18.413566    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw"
Nov 02 01:45:18.428940 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:18.428915    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-xgfxp"
Nov 02 01:45:18.517417 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:18.517386    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c86d89a-15d9-4aa0-86be-a24ad9ea29de-kube-api-access-fb9bg\" (UniqueName: \"kubernetes.io/projected/6c86d89a-15d9-4aa0-86be-a24ad9ea29de-kube-api-access-fb9bg\") pod \"6c86d89a-15d9-4aa0-86be-a24ad9ea29de\" (UID: \"6c86d89a-15d9-4aa0-86be-a24ad9ea29de\") "
Nov 02 01:45:18.517569 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:18.517428    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7950aea7-d901-478a-a27e-f5798c935c66-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7950aea7-d901-478a-a27e-f5798c935c66-maas-api-service-ca\") pod \"7950aea7-d901-478a-a27e-f5798c935c66\" (UID: \"7950aea7-d901-478a-a27e-f5798c935c66\") "
Nov 02 01:45:18.517569 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:18.517498    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c86d89a-15d9-4aa0-86be-a24ad9ea29de-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c86d89a-15d9-4aa0-86be-a24ad9ea29de-maas-api-service-ca\") pod \"6c86d89a-15d9-4aa0-86be-a24ad9ea29de\" (UID: \"6c86d89a-15d9-4aa0-86be-a24ad9ea29de\") "
Nov 02 01:45:18.517714 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:18.517598    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7950aea7-d901-478a-a27e-f5798c935c66-kube-api-access-4gcwx\" (UniqueName: \"kubernetes.io/projected/7950aea7-d901-478a-a27e-f5798c935c66-kube-api-access-4gcwx\") pod \"7950aea7-d901-478a-a27e-f5798c935c66\" (UID: \"7950aea7-d901-478a-a27e-f5798c935c66\") "
Nov 02 01:45:18.517924 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:18.517832    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7950aea7-d901-478a-a27e-f5798c935c66-maas-api-service-ca" pod "7950aea7-d901-478a-a27e-f5798c935c66" (UID: "7950aea7-d901-478a-a27e-f5798c935c66"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:18.517924 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:18.517895    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c86d89a-15d9-4aa0-86be-a24ad9ea29de-maas-api-service-ca" pod "6c86d89a-15d9-4aa0-86be-a24ad9ea29de" (UID: "6c86d89a-15d9-4aa0-86be-a24ad9ea29de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:18.518331 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:18.518304    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7950aea7-d901-478a-a27e-f5798c935c66-maas-api-service-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:45:18.518622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:18.518334    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c86d89a-15d9-4aa0-86be-a24ad9ea29de-maas-api-service-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:45:18.519937 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:18.519911    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c86d89a-15d9-4aa0-86be-a24ad9ea29de-kube-api-access-fb9bg" pod "6c86d89a-15d9-4aa0-86be-a24ad9ea29de" (UID: "6c86d89a-15d9-4aa0-86be-a24ad9ea29de"). InnerVolumeSpecName "kube-api-access-fb9bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:18.520098 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:18.520076    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7950aea7-d901-478a-a27e-f5798c935c66-kube-api-access-4gcwx" pod "7950aea7-d901-478a-a27e-f5798c935c66" (UID: "7950aea7-d901-478a-a27e-f5798c935c66"). InnerVolumeSpecName "kube-api-access-4gcwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:18.620230 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:18.620196    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb9bg\" (UniqueName: \"kubernetes.io/projected/6c86d89a-15d9-4aa0-86be-a24ad9ea29de-kube-api-access-fb9bg\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:45:18.620230 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:18.620225    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gcwx\" (UniqueName: \"kubernetes.io/projected/7950aea7-d901-478a-a27e-f5798c935c66-kube-api-access-4gcwx\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:45:19.219143 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:19.219110    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc3de286-fa4a-4ff9-be08-163ffb17d034" path="/var/lib/kubelet/pods/bc3de286-fa4a-4ff9-be08-163ffb17d034/volumes"
Nov 02 01:45:19.248336 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:19.248274    5639 generic.go:356] "Generic (PLEG): container finished" podID="a73a71e7-0590-4e7f-aa7a-059d7b53b75a" containerID="76806f8cb07bf9d5db3e545ef80fe4be4184e0a42cfee9a3c931f8bbe12084e9" exitCode=60
Nov 02 01:45:19.248503 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:19.248353    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw" event={"ID":"a73a71e7-0590-4e7f-aa7a-059d7b53b75a","Type":"ContainerDied","Data":"76806f8cb07bf9d5db3e545ef80fe4be4184e0a42cfee9a3c931f8bbe12084e9"}
Nov 02 01:45:19.248503 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:19.248414    5639 scope.go:122] "RemoveContainer" containerID="3a7bc99ac08bb7756e70a4b27d3649ce242c1a6b5134fd43b66db6688fd1b6b6"
Nov 02 01:45:19.248821 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:19.248803    5639 scope.go:122] "RemoveContainer" containerID="76806f8cb07bf9d5db3e545ef80fe4be4184e0a42cfee9a3c931f8bbe12084e9"
Nov 02 01:45:19.249176 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:45:19.249149    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw_odh-ai-gateway-infra(a73a71e7-0590-4e7f-aa7a-059d7b53b75a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw" podUID="a73a71e7-0590-4e7f-aa7a-059d7b53b75a"
Nov 02 01:45:19.249915 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:19.249899    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw"
Nov 02 01:45:19.249989 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:19.249909    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw" event={"ID":"7950aea7-d901-478a-a27e-f5798c935c66","Type":"ContainerDied","Data":"20e9b1262fdbea1d2f752a9e0797a04c4afff04fa04abb01c0aafcf1ad5e43dc"}
Nov 02 01:45:19.251420 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:19.251399    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-xgfxp"
Nov 02 01:45:19.251531 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:19.251429    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-xgfxp" event={"ID":"6c86d89a-15d9-4aa0-86be-a24ad9ea29de","Type":"ContainerDied","Data":"223306c1e9b280850adf16d9a1b658e96e78e3b484b4dfaaec709d11e1187ccf"}
Nov 02 01:45:19.261333 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:19.261311    5639 scope.go:122] "RemoveContainer" containerID="c1481e673299d9754f204395435f89d285fb39a33002d35e2a16a97f98a58786"
Nov 02 01:45:19.271067 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:19.271047    5639 scope.go:122] "RemoveContainer" containerID="0edf4887103f4f4fc3479b8eaee07fe1b95fbf8c320df3f2db7f5ef4e26caa5b"
Nov 02 01:45:19.470717 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:19.470677    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf"]
Nov 02 01:45:19.475966 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:19.475936    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw"]
Nov 02 01:45:19.481028 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:19.480952    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893065-wxlmw"]
Nov 02 01:45:19.591696 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:19.591664    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893065-xgfxp"]
Nov 02 01:45:19.595634 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:19.595604    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893065-xgfxp"]
Nov 02 01:45:19.633509 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:19.633477    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf"
Nov 02 01:45:19.734403 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:19.734269    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/509e47f7-b40b-4e9e-98be-d13fe4cb99c8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/509e47f7-b40b-4e9e-98be-d13fe4cb99c8-maas-api-service-ca\") pod \"509e47f7-b40b-4e9e-98be-d13fe4cb99c8\" (UID: \"509e47f7-b40b-4e9e-98be-d13fe4cb99c8\") "
Nov 02 01:45:19.734543 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:19.734478    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/509e47f7-b40b-4e9e-98be-d13fe4cb99c8-kube-api-access-7bbv4\" (UniqueName: \"kubernetes.io/projected/509e47f7-b40b-4e9e-98be-d13fe4cb99c8-kube-api-access-7bbv4\") pod \"509e47f7-b40b-4e9e-98be-d13fe4cb99c8\" (UID: \"509e47f7-b40b-4e9e-98be-d13fe4cb99c8\") "
Nov 02 01:45:19.734643 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:19.734621    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/509e47f7-b40b-4e9e-98be-d13fe4cb99c8-maas-api-service-ca" pod "509e47f7-b40b-4e9e-98be-d13fe4cb99c8" (UID: "509e47f7-b40b-4e9e-98be-d13fe4cb99c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:19.735002 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:19.734985    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/509e47f7-b40b-4e9e-98be-d13fe4cb99c8-maas-api-service-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:45:19.736733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:19.736711    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/509e47f7-b40b-4e9e-98be-d13fe4cb99c8-kube-api-access-7bbv4" pod "509e47f7-b40b-4e9e-98be-d13fe4cb99c8" (UID: "509e47f7-b40b-4e9e-98be-d13fe4cb99c8"). InnerVolumeSpecName "kube-api-access-7bbv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:19.836136 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:19.836104    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bbv4\" (UniqueName: \"kubernetes.io/projected/509e47f7-b40b-4e9e-98be-d13fe4cb99c8-kube-api-access-7bbv4\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:45:20.260410 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:20.260379    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf"
Nov 02 01:45:20.260410 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:20.260399    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf" event={"ID":"509e47f7-b40b-4e9e-98be-d13fe4cb99c8","Type":"ContainerDied","Data":"50997caed3e19f90409458b465e74034fe31f12f7f5cef364feb29121b87f164"}
Nov 02 01:45:20.260906 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:20.260467    5639 scope.go:122] "RemoveContainer" containerID="bc7267a6398a8fce28f06fe779c2c23447aa345bf7352183dc91835d22d062c7"
Nov 02 01:45:20.374190 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:20.374159    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw"]
Nov 02 01:45:20.425600 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:20.425566    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf"]
Nov 02 01:45:20.431763 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:20.431735    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893065-w4hrf"]
Nov 02 01:45:20.528827 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:20.528753    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw"
Nov 02 01:45:20.645976 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:20.645941    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a73a71e7-0590-4e7f-aa7a-059d7b53b75a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a73a71e7-0590-4e7f-aa7a-059d7b53b75a-maas-api-service-ca\") pod \"a73a71e7-0590-4e7f-aa7a-059d7b53b75a\" (UID: \"a73a71e7-0590-4e7f-aa7a-059d7b53b75a\") "
Nov 02 01:45:20.645976 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:20.645982    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a73a71e7-0590-4e7f-aa7a-059d7b53b75a-kube-api-access-lmcqj\" (UniqueName: \"kubernetes.io/projected/a73a71e7-0590-4e7f-aa7a-059d7b53b75a-kube-api-access-lmcqj\") pod \"a73a71e7-0590-4e7f-aa7a-059d7b53b75a\" (UID: \"a73a71e7-0590-4e7f-aa7a-059d7b53b75a\") "
Nov 02 01:45:20.646382 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:20.646331    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a73a71e7-0590-4e7f-aa7a-059d7b53b75a-maas-api-service-ca" pod "a73a71e7-0590-4e7f-aa7a-059d7b53b75a" (UID: "a73a71e7-0590-4e7f-aa7a-059d7b53b75a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:20.646560 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:20.646546    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a73a71e7-0590-4e7f-aa7a-059d7b53b75a-maas-api-service-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:45:20.648237 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:20.648215    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a73a71e7-0590-4e7f-aa7a-059d7b53b75a-kube-api-access-lmcqj" pod "a73a71e7-0590-4e7f-aa7a-059d7b53b75a" (UID: "a73a71e7-0590-4e7f-aa7a-059d7b53b75a"). InnerVolumeSpecName "kube-api-access-lmcqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:20.747934 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:20.747888    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmcqj\" (UniqueName: \"kubernetes.io/projected/a73a71e7-0590-4e7f-aa7a-059d7b53b75a-kube-api-access-lmcqj\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:45:21.220334 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:21.220272    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="509e47f7-b40b-4e9e-98be-d13fe4cb99c8" path="/var/lib/kubelet/pods/509e47f7-b40b-4e9e-98be-d13fe4cb99c8/volumes"
Nov 02 01:45:21.220752 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:21.220739    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c86d89a-15d9-4aa0-86be-a24ad9ea29de" path="/var/lib/kubelet/pods/6c86d89a-15d9-4aa0-86be-a24ad9ea29de/volumes"
Nov 02 01:45:21.221073 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:21.221062    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7950aea7-d901-478a-a27e-f5798c935c66" path="/var/lib/kubelet/pods/7950aea7-d901-478a-a27e-f5798c935c66/volumes"
Nov 02 01:45:21.274236 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:21.274208    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw"
Nov 02 01:45:21.274709 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:21.274235    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw" event={"ID":"a73a71e7-0590-4e7f-aa7a-059d7b53b75a","Type":"ContainerDied","Data":"ff005c5432cac209ae615ba8c0c498915e2553ce52cc8b1a4d77e8f57ad43860"}
Nov 02 01:45:21.274709 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:21.274312    5639 scope.go:122] "RemoveContainer" containerID="76806f8cb07bf9d5db3e545ef80fe4be4184e0a42cfee9a3c931f8bbe12084e9"
Nov 02 01:45:21.438791 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:21.438738    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw"]
Nov 02 01:45:21.442571 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:21.442546    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893065-z4lzw"]
Nov 02 01:45:23.219835 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:23.219802    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a73a71e7-0590-4e7f-aa7a-059d7b53b75a" path="/var/lib/kubelet/pods/a73a71e7-0590-4e7f-aa7a-059d7b53b75a/volumes"
Nov 02 01:45:25.701733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:45:25.701705    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:45:27.427700 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:45:27.427650    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:45:57.462627 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:45:57.462534    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:46:07.723888 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:07.723853    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:46:09.477973 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.477938    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"]
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.479099    5639 cpu_manager.go:414] "containerMap: removing container" podUID="6c86d89a-15d9-4aa0-86be-a24ad9ea29de" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.479132    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c86d89a-15d9-4aa0-86be-a24ad9ea29de" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.479164    5639 cpu_manager.go:414] "containerMap: removing container" podUID="a73a71e7-0590-4e7f-aa7a-059d7b53b75a" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.479172    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a73a71e7-0590-4e7f-aa7a-059d7b53b75a" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.479193    5639 cpu_manager.go:414] "containerMap: removing container" podUID="bc3de286-fa4a-4ff9-be08-163ffb17d034" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.479201    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc3de286-fa4a-4ff9-be08-163ffb17d034" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.479235    5639 cpu_manager.go:414] "containerMap: removing container" podUID="7950aea7-d901-478a-a27e-f5798c935c66" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.479243    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7950aea7-d901-478a-a27e-f5798c935c66" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.479325    5639 cpu_manager.go:414] "containerMap: removing container" podUID="509e47f7-b40b-4e9e-98be-d13fe4cb99c8" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.479334    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="509e47f7-b40b-4e9e-98be-d13fe4cb99c8" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.479355    5639 cpu_manager.go:414] "containerMap: removing container" podUID="509e47f7-b40b-4e9e-98be-d13fe4cb99c8" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.479362    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="509e47f7-b40b-4e9e-98be-d13fe4cb99c8" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.479393    5639 cpu_manager.go:414] "containerMap: removing container" podUID="a73a71e7-0590-4e7f-aa7a-059d7b53b75a" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.479400    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a73a71e7-0590-4e7f-aa7a-059d7b53b75a" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.479431    5639 cpu_manager.go:414] "containerMap: removing container" podUID="bc3de286-fa4a-4ff9-be08-163ffb17d034" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.479438    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc3de286-fa4a-4ff9-be08-163ffb17d034" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.479465    5639 cpu_manager.go:414] "containerMap: removing container" podUID="7950aea7-d901-478a-a27e-f5798c935c66" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.479471    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7950aea7-d901-478a-a27e-f5798c935c66" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.479520    5639 cpu_manager.go:414] "containerMap: removing container" podUID="bc3de286-fa4a-4ff9-be08-163ffb17d034" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.479529    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc3de286-fa4a-4ff9-be08-163ffb17d034" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.479570    5639 cpu_manager.go:414] "containerMap: removing container" podUID="6c86d89a-15d9-4aa0-86be-a24ad9ea29de" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.479577    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c86d89a-15d9-4aa0-86be-a24ad9ea29de" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.479955    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c86d89a-15d9-4aa0-86be-a24ad9ea29de" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.479981    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="a73a71e7-0590-4e7f-aa7a-059d7b53b75a" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.479992    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc3de286-fa4a-4ff9-be08-163ffb17d034" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.480009    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="7950aea7-d901-478a-a27e-f5798c935c66" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.480019    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="a73a71e7-0590-4e7f-aa7a-059d7b53b75a" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.480036    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="509e47f7-b40b-4e9e-98be-d13fe4cb99c8" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.480052    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="509e47f7-b40b-4e9e-98be-d13fe4cb99c8" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.480063    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="a73a71e7-0590-4e7f-aa7a-059d7b53b75a" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.480081    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc3de286-fa4a-4ff9-be08-163ffb17d034" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.480096    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="509e47f7-b40b-4e9e-98be-d13fe4cb99c8" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.480105    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc3de286-fa4a-4ff9-be08-163ffb17d034" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.480125    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="7950aea7-d901-478a-a27e-f5798c935c66" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.480132    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c86d89a-15d9-4aa0-86be-a24ad9ea29de" containerName="cleanup"
Nov 02 01:46:09.486623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.485382    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.536152 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.536116    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.536355 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.536161    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.536355 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.536188    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.536355 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.536325    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.536502 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.536356    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.536502 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.536385    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.536502 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.536468    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.536502 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.536497    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.536750 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.536537    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qb72\" (UniqueName: \"kubernetes.io/projected/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-kube-api-access-2qb72\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.591597 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.591562    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"]
Nov 02 01:46:09.637715 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.637681    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.637911 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.637732    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.637911 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.637762    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.637911 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.637791    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.637911 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.637810    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.637911 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.637848    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.637911 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.637889    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.638355 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.637921    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.638355 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.637982    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qb72\" (UniqueName: \"kubernetes.io/projected/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-kube-api-access-2qb72\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.638355 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.638264    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.638578 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.638554    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.639420 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.639395    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.639547 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.639421    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.639547 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.639477    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.640382 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.640360    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.640693 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.640676    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.699364 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.699337    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.699506 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.699339    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qb72\" (UniqueName: \"kubernetes.io/projected/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-kube-api-access-2qb72\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:09.800868 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:09.800788    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:10.059664 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:10.059578    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4be80e0c-dcsxz"]
Nov 02 01:46:10.062956 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:10.062929    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4be80e0c-dcsxz"]
Nov 02 01:46:10.233421 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:46:10.233392    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4443617_f4c0_40c4_a3bf_0b6241d6ce97.slice/crio-8ce894c17459853e2c06ec6f05e9dc967a0215e1cd84c6050c240a6d4ad3b141 WatchSource:0}: Error finding container 8ce894c17459853e2c06ec6f05e9dc967a0215e1cd84c6050c240a6d4ad3b141: Status 404 returned error can't find the container with id 8ce894c17459853e2c06ec6f05e9dc967a0215e1cd84c6050c240a6d4ad3b141
Nov 02 01:46:10.234340 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:10.234309    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"]
Nov 02 01:46:10.235971 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:10.235942    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:46:10.236047 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:10.236004    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:46:10.236047 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:10.236033    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:46:10.646830 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:10.646788    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f" event={"ID":"c4443617-f4c0-40c4-a3bf-0b6241d6ce97","Type":"ContainerStarted","Data":"4cc52e35c4e0543b16494f86403df9c229aeea264497edd5c8cf3b296e658725"}
Nov 02 01:46:10.647248 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:10.646853    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f" event={"ID":"c4443617-f4c0-40c4-a3bf-0b6241d6ce97","Type":"ContainerStarted","Data":"8ce894c17459853e2c06ec6f05e9dc967a0215e1cd84c6050c240a6d4ad3b141"}
Nov 02 01:46:10.769431 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:10.769368    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f" podStartSLOduration=1.769336543 podStartE2EDuration="1.769336543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:46:10.767784127 +0000 UTC m=+1235.671466243" watchObservedRunningTime="2026-11-02 01:46:10.769336543 +0000 UTC m=+1235.673018657"
Nov 02 01:46:10.801540 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:10.801509    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:11.219505 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:11.219475    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5605bbd9-39a9-436d-a4c6-d0a73900162e" path="/var/lib/kubelet/pods/5605bbd9-39a9-436d-a4c6-d0a73900162e/volumes"
Nov 02 01:46:11.802825 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:11.802788    5639 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 01:46:11.803252 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:11.802843    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f" podUID="c4443617-f4c0-40c4-a3bf-0b6241d6ce97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 01:46:12.801716 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:12.801676    5639 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 01:46:12.801911 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:12.801779    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f" podUID="c4443617-f4c0-40c4-a3bf-0b6241d6ce97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 01:46:13.801565 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:13.801531    5639 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 01:46:13.801972 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:13.801586    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f" podUID="c4443617-f4c0-40c4-a3bf-0b6241d6ce97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 01:46:14.801472 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:14.801436    5639 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 01:46:14.801663 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:14.801493    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f" podUID="c4443617-f4c0-40c4-a3bf-0b6241d6ce97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 01:46:15.801905 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:15.801867    5639 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 01:46:15.802454 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:15.801926    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f" podUID="c4443617-f4c0-40c4-a3bf-0b6241d6ce97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 01:46:16.801848 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:16.801812    5639 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 01:46:16.802147 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:16.802115    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f" podUID="c4443617-f4c0-40c4-a3bf-0b6241d6ce97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 01:46:17.667989 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.667947    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"]
Nov 02 01:46:17.668627 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.668608    5639 cpu_manager.go:414] "containerMap: removing container" podUID="6c86d89a-15d9-4aa0-86be-a24ad9ea29de" containerName="cleanup"
Nov 02 01:46:17.668627 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.668628    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c86d89a-15d9-4aa0-86be-a24ad9ea29de" containerName="cleanup"
Nov 02 01:46:17.668795 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.668669    5639 cpu_manager.go:414] "containerMap: removing container" podUID="509e47f7-b40b-4e9e-98be-d13fe4cb99c8" containerName="cleanup"
Nov 02 01:46:17.668795 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.668678    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="509e47f7-b40b-4e9e-98be-d13fe4cb99c8" containerName="cleanup"
Nov 02 01:46:17.668795 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.668706    5639 cpu_manager.go:414] "containerMap: removing container" podUID="a73a71e7-0590-4e7f-aa7a-059d7b53b75a" containerName="cleanup"
Nov 02 01:46:17.668795 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.668713    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a73a71e7-0590-4e7f-aa7a-059d7b53b75a" containerName="cleanup"
Nov 02 01:46:17.668795 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.668735    5639 cpu_manager.go:414] "containerMap: removing container" podUID="7950aea7-d901-478a-a27e-f5798c935c66" containerName="cleanup"
Nov 02 01:46:17.668795 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.668742    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7950aea7-d901-478a-a27e-f5798c935c66" containerName="cleanup"
Nov 02 01:46:17.669080 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.668905    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c86d89a-15d9-4aa0-86be-a24ad9ea29de" containerName="cleanup"
Nov 02 01:46:17.669080 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.668924    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="7950aea7-d901-478a-a27e-f5798c935c66" containerName="cleanup"
Nov 02 01:46:17.672728 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.672705    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:46:17.775725 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.775672    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e06ea2\"/\"openshift-service-ca.crt\""
Nov 02 01:46:17.775725 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.775698    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e06ea2\"/\"kube-root-ca.crt\""
Nov 02 01:46:17.775725 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.775687    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e06ea2\"/\"default-dockercfg-mqqzb\""
Nov 02 01:46:17.775985 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.775797    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e06ea2\"/\"distinct-2-w5-e06ea2-kserve-self-signed-certs\""
Nov 02 01:46:17.779103 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.779080    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"]
Nov 02 01:46:17.782377 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.782353    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5bl4\" (UniqueName: \"kubernetes.io/projected/fad03279-07f7-4f16-852c-b9c8fb6456dd-kube-api-access-k5bl4\") pod \"distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92\" (UID: \"fad03279-07f7-4f16-852c-b9c8fb6456dd\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:46:17.782442 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.782391    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-model-cache\") pod \"distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92\" (UID: \"fad03279-07f7-4f16-852c-b9c8fb6456dd\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:46:17.782564 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.782543    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-home\") pod \"distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92\" (UID: \"fad03279-07f7-4f16-852c-b9c8fb6456dd\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:46:17.782600 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.782575    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-tmp-dir\") pod \"distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92\" (UID: \"fad03279-07f7-4f16-852c-b9c8fb6456dd\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:46:17.782645 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.782597    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fad03279-07f7-4f16-852c-b9c8fb6456dd-tls-certs\") pod \"distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92\" (UID: \"fad03279-07f7-4f16-852c-b9c8fb6456dd\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:46:17.782702 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.782688    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-dshm\") pod \"distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92\" (UID: \"fad03279-07f7-4f16-852c-b9c8fb6456dd\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:46:17.806047 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.806010    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:17.806504 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.806407    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:17.807089 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.807068    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 01:46:17.884143 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.884099    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-home\") pod \"distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92\" (UID: \"fad03279-07f7-4f16-852c-b9c8fb6456dd\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:46:17.885170 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.884788    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-home\") pod \"distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92\" (UID: \"fad03279-07f7-4f16-852c-b9c8fb6456dd\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:46:17.885374 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.885132    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-tmp-dir\") pod \"distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92\" (UID: \"fad03279-07f7-4f16-852c-b9c8fb6456dd\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:46:17.889447 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.889413    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fad03279-07f7-4f16-852c-b9c8fb6456dd-tls-certs\") pod \"distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92\" (UID: \"fad03279-07f7-4f16-852c-b9c8fb6456dd\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:46:17.891119 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.891093    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-dshm\") pod \"distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92\" (UID: \"fad03279-07f7-4f16-852c-b9c8fb6456dd\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:46:17.891623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.891602    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5bl4\" (UniqueName: \"kubernetes.io/projected/fad03279-07f7-4f16-852c-b9c8fb6456dd-kube-api-access-k5bl4\") pod \"distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92\" (UID: \"fad03279-07f7-4f16-852c-b9c8fb6456dd\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:46:17.892446 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.892424    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-model-cache\") pod \"distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92\" (UID: \"fad03279-07f7-4f16-852c-b9c8fb6456dd\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:46:17.893163 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.885810    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-tmp-dir\") pod \"distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92\" (UID: \"fad03279-07f7-4f16-852c-b9c8fb6456dd\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:46:17.894027 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.894003    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-model-cache\") pod \"distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92\" (UID: \"fad03279-07f7-4f16-852c-b9c8fb6456dd\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:46:17.909159 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.906599    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fad03279-07f7-4f16-852c-b9c8fb6456dd-tls-certs\") pod \"distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92\" (UID: \"fad03279-07f7-4f16-852c-b9c8fb6456dd\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:46:17.909159 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.908919    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-dshm\") pod \"distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92\" (UID: \"fad03279-07f7-4f16-852c-b9c8fb6456dd\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:46:17.962350 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.958674    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5bl4\" (UniqueName: \"kubernetes.io/projected/fad03279-07f7-4f16-852c-b9c8fb6456dd-kube-api-access-k5bl4\") pod \"distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92\" (UID: \"fad03279-07f7-4f16-852c-b9c8fb6456dd\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:46:17.987691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:17.987654    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:46:18.380490 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:18.380464    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"]
Nov 02 01:46:18.381375 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:46:18.381342    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfad03279_07f7_4f16_852c_b9c8fb6456dd.slice/crio-ce4d435461dadadbeb5a42e21d301f67635957bbfe0e408d7b5728cdc38650ce WatchSource:0}: Error finding container ce4d435461dadadbeb5a42e21d301f67635957bbfe0e408d7b5728cdc38650ce: Status 404 returned error can't find the container with id ce4d435461dadadbeb5a42e21d301f67635957bbfe0e408d7b5728cdc38650ce
Nov 02 01:46:18.732464 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:18.732358    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" event={"ID":"fad03279-07f7-4f16-852c-b9c8fb6456dd","Type":"ContainerStarted","Data":"9d8c1830785e6186525eb2eba18cc0e4fa40165f338b420426268500da3f00f7"}
Nov 02 01:46:18.732464 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:18.732409    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" event={"ID":"fad03279-07f7-4f16-852c-b9c8fb6456dd","Type":"ContainerStarted","Data":"ce4d435461dadadbeb5a42e21d301f67635957bbfe0e408d7b5728cdc38650ce"}
Nov 02 01:46:18.851440 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:18.851388    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podStartSLOduration=1.851374552 podStartE2EDuration="1.851374552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:46:18.848592332 +0000 UTC m=+1243.752274449" watchObservedRunningTime="2026-11-02 01:46:18.851374552 +0000 UTC m=+1243.755056666"
Nov 02 01:46:18.992361 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:18.990584    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:46:27.424640 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:46:27.424598    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:46:27.987846 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:27.987801    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:46:28.002355 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:28.002326    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:46:28.836621 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:28.836592    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:46:37.219743 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:37.219712    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffa5e8cd-8654k"]
Nov 02 01:46:37.223936 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:37.223917    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa5e8cd-8654k"
Nov 02 01:46:37.275893 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:37.275863    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffa5e8cd-8654k"]
Nov 02 01:46:37.276094 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:37.276004    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:46:37.276094 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:37.276061    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:46:37.321035 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:37.320992    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:46:37.321035 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:37.320992    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:46:37.364421 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:37.364397    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/febf7f8f-3020-4264-bab0-4e25622b2696-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffa5e8cd-8654k\" (UID: \"febf7f8f-3020-4264-bab0-4e25622b2696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa5e8cd-8654k"
Nov 02 01:46:37.364558 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:37.364456    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/febf7f8f-3020-4264-bab0-4e25622b2696-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffa5e8cd-8654k\" (UID: \"febf7f8f-3020-4264-bab0-4e25622b2696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa5e8cd-8654k"
Nov 02 01:46:37.364558 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:37.364536    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2bjn\" (UniqueName: \"kubernetes.io/projected/febf7f8f-3020-4264-bab0-4e25622b2696-kube-api-access-m2bjn\") pod \"maas-api-revoke-sub-ffa5e8cd-8654k\" (UID: \"febf7f8f-3020-4264-bab0-4e25622b2696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa5e8cd-8654k"
Nov 02 01:46:37.466103 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:37.466070    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/febf7f8f-3020-4264-bab0-4e25622b2696-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffa5e8cd-8654k\" (UID: \"febf7f8f-3020-4264-bab0-4e25622b2696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa5e8cd-8654k"
Nov 02 01:46:37.466272 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:37.466123    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2bjn\" (UniqueName: \"kubernetes.io/projected/febf7f8f-3020-4264-bab0-4e25622b2696-kube-api-access-m2bjn\") pod \"maas-api-revoke-sub-ffa5e8cd-8654k\" (UID: \"febf7f8f-3020-4264-bab0-4e25622b2696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa5e8cd-8654k"
Nov 02 01:46:37.466390 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:37.466372    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/febf7f8f-3020-4264-bab0-4e25622b2696-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffa5e8cd-8654k\" (UID: \"febf7f8f-3020-4264-bab0-4e25622b2696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa5e8cd-8654k"
Nov 02 01:46:37.467030 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:37.466992    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/febf7f8f-3020-4264-bab0-4e25622b2696-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffa5e8cd-8654k\" (UID: \"febf7f8f-3020-4264-bab0-4e25622b2696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa5e8cd-8654k"
Nov 02 01:46:37.468692 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:37.468663    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/febf7f8f-3020-4264-bab0-4e25622b2696-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffa5e8cd-8654k\" (UID: \"febf7f8f-3020-4264-bab0-4e25622b2696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa5e8cd-8654k"
Nov 02 01:46:37.527926 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:37.527787    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2bjn\" (UniqueName: \"kubernetes.io/projected/febf7f8f-3020-4264-bab0-4e25622b2696-kube-api-access-m2bjn\") pod \"maas-api-revoke-sub-ffa5e8cd-8654k\" (UID: \"febf7f8f-3020-4264-bab0-4e25622b2696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa5e8cd-8654k"
Nov 02 01:46:37.539673 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:37.539646    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa5e8cd-8654k"
Nov 02 01:46:37.927325 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:37.927296    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffa5e8cd-8654k"]
Nov 02 01:46:38.903451 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:38.903413    5639 generic.go:356] "Generic (PLEG): container finished" podID="febf7f8f-3020-4264-bab0-4e25622b2696" containerID="634d515bd66acba64c7daca22ac3d869a45236cf37435732c68ab70b4a3ca255" exitCode=0
Nov 02 01:46:38.903867 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:38.903499    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa5e8cd-8654k" event={"ID":"febf7f8f-3020-4264-bab0-4e25622b2696","Type":"ContainerDied","Data":"634d515bd66acba64c7daca22ac3d869a45236cf37435732c68ab70b4a3ca255"}
Nov 02 01:46:38.903867 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:38.903547    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa5e8cd-8654k" event={"ID":"febf7f8f-3020-4264-bab0-4e25622b2696","Type":"ContainerStarted","Data":"8304c183e0dc5d1d0ecab96a4f21319e962b656b94eb3fc025b12885b2c29b28"}
Nov 02 01:46:39.074526 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:39.074491    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08942016-s2qzs"]
Nov 02 01:46:39.077861 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:39.077837    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08942016-s2qzs"]
Nov 02 01:46:39.228458 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:39.228396    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8657d1b-7690-40be-b28a-a13e6633bb6b" path="/var/lib/kubelet/pods/f8657d1b-7690-40be-b28a-a13e6633bb6b/volumes"
Nov 02 01:46:40.069404 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:40.069375    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa5e8cd-8654k"
Nov 02 01:46:40.225219 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:40.225103    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/febf7f8f-3020-4264-bab0-4e25622b2696-kube-api-access-m2bjn\" (UniqueName: \"kubernetes.io/projected/febf7f8f-3020-4264-bab0-4e25622b2696-kube-api-access-m2bjn\") pod \"febf7f8f-3020-4264-bab0-4e25622b2696\" (UID: \"febf7f8f-3020-4264-bab0-4e25622b2696\") "
Nov 02 01:46:40.225415 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:40.225232    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/febf7f8f-3020-4264-bab0-4e25622b2696-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/febf7f8f-3020-4264-bab0-4e25622b2696-maas-api-xks-ca\") pod \"febf7f8f-3020-4264-bab0-4e25622b2696\" (UID: \"febf7f8f-3020-4264-bab0-4e25622b2696\") "
Nov 02 01:46:40.225415 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:40.225399    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/febf7f8f-3020-4264-bab0-4e25622b2696-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/febf7f8f-3020-4264-bab0-4e25622b2696-maas-api-openshift-ca\") pod \"febf7f8f-3020-4264-bab0-4e25622b2696\" (UID: \"febf7f8f-3020-4264-bab0-4e25622b2696\") "
Nov 02 01:46:40.225853 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:40.225821    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/febf7f8f-3020-4264-bab0-4e25622b2696-maas-api-openshift-ca" pod "febf7f8f-3020-4264-bab0-4e25622b2696" (UID: "febf7f8f-3020-4264-bab0-4e25622b2696"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:46:40.226097 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:40.226010    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/febf7f8f-3020-4264-bab0-4e25622b2696-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:46:40.227587 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:40.227563    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/febf7f8f-3020-4264-bab0-4e25622b2696-maas-api-xks-ca" pod "febf7f8f-3020-4264-bab0-4e25622b2696" (UID: "febf7f8f-3020-4264-bab0-4e25622b2696"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:46:40.227884 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:40.227861    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/febf7f8f-3020-4264-bab0-4e25622b2696-kube-api-access-m2bjn" pod "febf7f8f-3020-4264-bab0-4e25622b2696" (UID: "febf7f8f-3020-4264-bab0-4e25622b2696"). InnerVolumeSpecName "kube-api-access-m2bjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:46:40.327634 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:40.327594    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2bjn\" (UniqueName: \"kubernetes.io/projected/febf7f8f-3020-4264-bab0-4e25622b2696-kube-api-access-m2bjn\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:46:40.327634 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:40.327627    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/febf7f8f-3020-4264-bab0-4e25622b2696-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:46:40.962468 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:40.961587    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa5e8cd-8654k" event={"ID":"febf7f8f-3020-4264-bab0-4e25622b2696","Type":"ContainerDied","Data":"8304c183e0dc5d1d0ecab96a4f21319e962b656b94eb3fc025b12885b2c29b28"}
Nov 02 01:46:40.962468 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:40.961654    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8304c183e0dc5d1d0ecab96a4f21319e962b656b94eb3fc025b12885b2c29b28"
Nov 02 01:46:40.962468 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:40.961728    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa5e8cd-8654k"
Nov 02 01:46:41.841928 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:41.841895    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:46:52.887904 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:52.887823    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f385a5f1-vkrgb"]
Nov 02 01:46:52.888469 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:52.888454    5639 cpu_manager.go:414] "containerMap: removing container" podUID="febf7f8f-3020-4264-bab0-4e25622b2696" containerName="revoke-keys"
Nov 02 01:46:52.888534 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:52.888470    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="febf7f8f-3020-4264-bab0-4e25622b2696" containerName="revoke-keys"
Nov 02 01:46:52.888569 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:52.888562    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="febf7f8f-3020-4264-bab0-4e25622b2696" containerName="revoke-keys"
Nov 02 01:46:52.892092 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:52.892072    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f385a5f1-vkrgb"
Nov 02 01:46:52.993767 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:52.993730    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:46:52.993767 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:52.993731    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:46:52.993767 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:52.993738    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:46:52.994044 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:52.993810    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:46:52.997349 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:52.997324    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f385a5f1-vkrgb"]
Nov 02 01:46:53.053975 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:53.053937    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ceca1f-fff5-4d36-9de2-aaf984edd17b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f385a5f1-vkrgb\" (UID: \"a3ceca1f-fff5-4d36-9de2-aaf984edd17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f385a5f1-vkrgb"
Nov 02 01:46:53.054153 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:53.053991    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ceca1f-fff5-4d36-9de2-aaf984edd17b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f385a5f1-vkrgb\" (UID: \"a3ceca1f-fff5-4d36-9de2-aaf984edd17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f385a5f1-vkrgb"
Nov 02 01:46:53.054153 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:53.054063    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6nn9\" (UniqueName: \"kubernetes.io/projected/a3ceca1f-fff5-4d36-9de2-aaf984edd17b-kube-api-access-z6nn9\") pod \"maas-api-revoke-sub-f385a5f1-vkrgb\" (UID: \"a3ceca1f-fff5-4d36-9de2-aaf984edd17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f385a5f1-vkrgb"
Nov 02 01:46:53.155215 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:53.155123    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ceca1f-fff5-4d36-9de2-aaf984edd17b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f385a5f1-vkrgb\" (UID: \"a3ceca1f-fff5-4d36-9de2-aaf984edd17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f385a5f1-vkrgb"
Nov 02 01:46:53.155215 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:53.155181    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ceca1f-fff5-4d36-9de2-aaf984edd17b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f385a5f1-vkrgb\" (UID: \"a3ceca1f-fff5-4d36-9de2-aaf984edd17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f385a5f1-vkrgb"
Nov 02 01:46:53.155215 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:53.155204    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6nn9\" (UniqueName: \"kubernetes.io/projected/a3ceca1f-fff5-4d36-9de2-aaf984edd17b-kube-api-access-z6nn9\") pod \"maas-api-revoke-sub-f385a5f1-vkrgb\" (UID: \"a3ceca1f-fff5-4d36-9de2-aaf984edd17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f385a5f1-vkrgb"
Nov 02 01:46:53.155925 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:53.155902    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ceca1f-fff5-4d36-9de2-aaf984edd17b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f385a5f1-vkrgb\" (UID: \"a3ceca1f-fff5-4d36-9de2-aaf984edd17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f385a5f1-vkrgb"
Nov 02 01:46:53.157730 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:53.157706    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ceca1f-fff5-4d36-9de2-aaf984edd17b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f385a5f1-vkrgb\" (UID: \"a3ceca1f-fff5-4d36-9de2-aaf984edd17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f385a5f1-vkrgb"
Nov 02 01:46:53.216465 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:53.216439    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6nn9\" (UniqueName: \"kubernetes.io/projected/a3ceca1f-fff5-4d36-9de2-aaf984edd17b-kube-api-access-z6nn9\") pod \"maas-api-revoke-sub-f385a5f1-vkrgb\" (UID: \"a3ceca1f-fff5-4d36-9de2-aaf984edd17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f385a5f1-vkrgb"
Nov 02 01:46:53.506724 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:53.506640    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f385a5f1-vkrgb"
Nov 02 01:46:53.883910 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:53.883877    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f385a5f1-vkrgb"]
Nov 02 01:46:53.884933 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:46:53.884902    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ceca1f_fff5_4d36_9de2_aaf984edd17b.slice/crio-a375ff2d1779c677e0a86d283ea7a81665dd58637acede8b1f848f8df5a5297d WatchSource:0}: Error finding container a375ff2d1779c677e0a86d283ea7a81665dd58637acede8b1f848f8df5a5297d: Status 404 returned error can't find the container with id a375ff2d1779c677e0a86d283ea7a81665dd58637acede8b1f848f8df5a5297d
Nov 02 01:46:54.076481 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:54.076452    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f385a5f1-vkrgb" event={"ID":"a3ceca1f-fff5-4d36-9de2-aaf984edd17b","Type":"ContainerStarted","Data":"a375ff2d1779c677e0a86d283ea7a81665dd58637acede8b1f848f8df5a5297d"}
Nov 02 01:46:55.048047 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.047968    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54fad367-ntrpm"]
Nov 02 01:46:55.085669 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.085639    5639 generic.go:356] "Generic (PLEG): container finished" podID="a3ceca1f-fff5-4d36-9de2-aaf984edd17b" containerID="326304970a3dd70a73795dd5ab7db92ae44eba237386820c7d2983c7f0ec001c" exitCode=0
Nov 02 01:46:55.086108 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.085726    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f385a5f1-vkrgb" event={"ID":"a3ceca1f-fff5-4d36-9de2-aaf984edd17b","Type":"ContainerDied","Data":"326304970a3dd70a73795dd5ab7db92ae44eba237386820c7d2983c7f0ec001c"}
Nov 02 01:46:55.120083 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.120050    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54fad367-ntrpm"]
Nov 02 01:46:55.219185 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.219154    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d4d547e-c9a3-45ea-a59b-90ecc0627947" path="/var/lib/kubelet/pods/0d4d547e-c9a3-45ea-a59b-90ecc0627947/volumes"
Nov 02 01:46:55.372995 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.372961    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"]
Nov 02 01:46:55.377225 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.377207    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.438640 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.438608    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"]
Nov 02 01:46:55.483208 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.483169    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.483457 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.483241    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.483457 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.483350    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.483600 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.483513    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.483600 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.483553    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.483715 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.483659    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.483715 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.483685    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz5mg\" (UniqueName: \"kubernetes.io/projected/cd05aad1-3b5f-43bd-988e-fce23728a36c-kube-api-access-gz5mg\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.483820 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.483718    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd05aad1-3b5f-43bd-988e-fce23728a36c-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.483820 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.483738    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.585615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.585579    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.585791 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.585634    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.585791 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.585658    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.585791 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.585736    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.585791 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.585753    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.585988 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.585815    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.585988 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.585957    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz5mg\" (UniqueName: \"kubernetes.io/projected/cd05aad1-3b5f-43bd-988e-fce23728a36c-kube-api-access-gz5mg\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.586086 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.586023    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd05aad1-3b5f-43bd-988e-fce23728a36c-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.586086 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.586036    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.586086 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.586056    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.586239 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.586096    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.586239 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.586141    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.586239 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.586177    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.587048 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.587021    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd05aad1-3b5f-43bd-988e-fce23728a36c-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.588506 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.588480    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.588717 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.588695    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.646973 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.646893    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.647176 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.647148    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz5mg\" (UniqueName: \"kubernetes.io/projected/cd05aad1-3b5f-43bd-988e-fce23728a36c-kube-api-access-gz5mg\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:55.692005 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:55.691958    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:56.130903 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:56.130874    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"]
Nov 02 01:46:56.132738 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:46:56.132710    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd05aad1_3b5f_43bd_988e_fce23728a36c.slice/crio-becba962892b10a8df494b74fd5d3e93de9bdcc73d3fb9fa8e68fef36af9077c WatchSource:0}: Error finding container becba962892b10a8df494b74fd5d3e93de9bdcc73d3fb9fa8e68fef36af9077c: Status 404 returned error can't find the container with id becba962892b10a8df494b74fd5d3e93de9bdcc73d3fb9fa8e68fef36af9077c
Nov 02 01:46:56.135756 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:56.135722    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:46:56.135860 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:56.135826    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:46:56.135922 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:56.135876    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:46:56.224735 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:56.224712    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f385a5f1-vkrgb"
Nov 02 01:46:56.294587 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:56.294553    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ceca1f-fff5-4d36-9de2-aaf984edd17b-kube-api-access-z6nn9\" (UniqueName: \"kubernetes.io/projected/a3ceca1f-fff5-4d36-9de2-aaf984edd17b-kube-api-access-z6nn9\") pod \"a3ceca1f-fff5-4d36-9de2-aaf984edd17b\" (UID: \"a3ceca1f-fff5-4d36-9de2-aaf984edd17b\") "
Nov 02 01:46:56.294783 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:56.294675    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3ceca1f-fff5-4d36-9de2-aaf984edd17b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ceca1f-fff5-4d36-9de2-aaf984edd17b-maas-api-openshift-ca\") pod \"a3ceca1f-fff5-4d36-9de2-aaf984edd17b\" (UID: \"a3ceca1f-fff5-4d36-9de2-aaf984edd17b\") "
Nov 02 01:46:56.294783 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:56.294729    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3ceca1f-fff5-4d36-9de2-aaf984edd17b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ceca1f-fff5-4d36-9de2-aaf984edd17b-maas-api-xks-ca\") pod \"a3ceca1f-fff5-4d36-9de2-aaf984edd17b\" (UID: \"a3ceca1f-fff5-4d36-9de2-aaf984edd17b\") "
Nov 02 01:46:56.295733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:56.295067    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3ceca1f-fff5-4d36-9de2-aaf984edd17b-maas-api-openshift-ca" pod "a3ceca1f-fff5-4d36-9de2-aaf984edd17b" (UID: "a3ceca1f-fff5-4d36-9de2-aaf984edd17b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:46:56.295733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:56.295697    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ceca1f-fff5-4d36-9de2-aaf984edd17b-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:46:56.297188 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:56.297160    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ceca1f-fff5-4d36-9de2-aaf984edd17b-kube-api-access-z6nn9" pod "a3ceca1f-fff5-4d36-9de2-aaf984edd17b" (UID: "a3ceca1f-fff5-4d36-9de2-aaf984edd17b"). InnerVolumeSpecName "kube-api-access-z6nn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:46:56.297382 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:56.297366    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3ceca1f-fff5-4d36-9de2-aaf984edd17b-maas-api-xks-ca" pod "a3ceca1f-fff5-4d36-9de2-aaf984edd17b" (UID: "a3ceca1f-fff5-4d36-9de2-aaf984edd17b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:46:56.396932 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:56.396854    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6nn9\" (UniqueName: \"kubernetes.io/projected/a3ceca1f-fff5-4d36-9de2-aaf984edd17b-kube-api-access-z6nn9\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:46:56.396932 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:56.396883    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ceca1f-fff5-4d36-9de2-aaf984edd17b-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:46:57.101305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:57.101238    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx" event={"ID":"cd05aad1-3b5f-43bd-988e-fce23728a36c","Type":"ContainerStarted","Data":"933c61e1d77c5c3d7b112435f828d16379c5e7e219797a482cc5c57783722be3"}
Nov 02 01:46:57.101502 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:57.101326    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx" event={"ID":"cd05aad1-3b5f-43bd-988e-fce23728a36c","Type":"ContainerStarted","Data":"becba962892b10a8df494b74fd5d3e93de9bdcc73d3fb9fa8e68fef36af9077c"}
Nov 02 01:46:57.110331 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:57.110261    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f385a5f1-vkrgb" event={"ID":"a3ceca1f-fff5-4d36-9de2-aaf984edd17b","Type":"ContainerDied","Data":"a375ff2d1779c677e0a86d283ea7a81665dd58637acede8b1f848f8df5a5297d"}
Nov 02 01:46:57.110474 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:57.110398    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a375ff2d1779c677e0a86d283ea7a81665dd58637acede8b1f848f8df5a5297d"
Nov 02 01:46:57.110557 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:57.110475    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f385a5f1-vkrgb"
Nov 02 01:46:57.290677 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:57.290609    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx" podStartSLOduration=2.290593127 podStartE2EDuration="2.290593127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:46:57.284872768 +0000 UTC m=+1282.188554882" watchObservedRunningTime="2026-11-02 01:46:57.290593127 +0000 UTC m=+1282.194275313"
Nov 02 01:46:57.422164 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:46:57.422083    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:46:57.693200 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:57.693115    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:46:57.694693 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:57.694668    5639 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 01:46:57.694815 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:57.694714    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx" podUID="cd05aad1-3b5f-43bd-988e-fce23728a36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 01:46:58.117038 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.117001    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"]
Nov 02 01:46:58.117937 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.117919    5639 cpu_manager.go:414] "containerMap: removing container" podUID="a3ceca1f-fff5-4d36-9de2-aaf984edd17b" containerName="revoke-keys"
Nov 02 01:46:58.117988 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.117941    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ceca1f-fff5-4d36-9de2-aaf984edd17b" containerName="revoke-keys"
Nov 02 01:46:58.118098 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.118087    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ceca1f-fff5-4d36-9de2-aaf984edd17b" containerName="revoke-keys"
Nov 02 01:46:58.122270 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.122251    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:46:58.193936 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.193906    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"]
Nov 02 01:46:58.221097 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.221070    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-tmp-dir\") pod \"distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5\" (UID: \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:46:58.221246 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.221135    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l87jl\" (UniqueName: \"kubernetes.io/projected/169d2fdb-6588-4d7e-9892-2d12c0e017b6-kube-api-access-l87jl\") pod \"distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5\" (UID: \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:46:58.221370 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.221345    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-model-cache\") pod \"distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5\" (UID: \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:46:58.221485 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.221468    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/169d2fdb-6588-4d7e-9892-2d12c0e017b6-tls-certs\") pod \"distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5\" (UID: \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:46:58.221530 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.221516    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-dshm\") pod \"distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5\" (UID: \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:46:58.221622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.221612    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-home\") pod \"distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5\" (UID: \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:46:58.239550 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.239525    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b71ba5\"/\"default-dockercfg-gxmsh\""
Nov 02 01:46:58.239695 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.239552    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b71ba5\"/\"openshift-service-ca.crt\""
Nov 02 01:46:58.239695 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.239627    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b71ba5\"/\"distinct-2-w0-b71ba5-kserve-self-signed-certs\""
Nov 02 01:46:58.239695 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.239635    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b71ba5\"/\"kube-root-ca.crt\""
Nov 02 01:46:58.327859 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.326100    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l87jl\" (UniqueName: \"kubernetes.io/projected/169d2fdb-6588-4d7e-9892-2d12c0e017b6-kube-api-access-l87jl\") pod \"distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5\" (UID: \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:46:58.327859 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.326191    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-model-cache\") pod \"distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5\" (UID: \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:46:58.331315 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.326321    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/169d2fdb-6588-4d7e-9892-2d12c0e017b6-tls-certs\") pod \"distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5\" (UID: \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:46:58.333675 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.331585    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-dshm\") pod \"distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5\" (UID: \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:46:58.334024 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.333993    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-home\") pod \"distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5\" (UID: \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:46:58.334203 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.334185    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-tmp-dir\") pod \"distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5\" (UID: \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:46:58.334631 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.334609    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-tmp-dir\") pod \"distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5\" (UID: \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:46:58.334853 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.329981    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-model-cache\") pod \"distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5\" (UID: \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:46:58.335948 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.335914    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-home\") pod \"distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5\" (UID: \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:46:58.338542 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.338517    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-dshm\") pod \"distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5\" (UID: \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:46:58.338671 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.338644    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/169d2fdb-6588-4d7e-9892-2d12c0e017b6-tls-certs\") pod \"distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5\" (UID: \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:46:58.390093 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.390003    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l87jl\" (UniqueName: \"kubernetes.io/projected/169d2fdb-6588-4d7e-9892-2d12c0e017b6-kube-api-access-l87jl\") pod \"distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5\" (UID: \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:46:58.435046 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.435014    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:46:58.693504 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.693381    5639 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 01:46:58.693680 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.693492    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx" podUID="cd05aad1-3b5f-43bd-988e-fce23728a36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 01:46:58.956328 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:46:58.956267    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod169d2fdb_6588_4d7e_9892_2d12c0e017b6.slice/crio-e3e4fac4aa265480c4e85b8806d366073b10965b6b277b0275105e9aa0ab39e9 WatchSource:0}: Error finding container e3e4fac4aa265480c4e85b8806d366073b10965b6b277b0275105e9aa0ab39e9: Status 404 returned error can't find the container with id e3e4fac4aa265480c4e85b8806d366073b10965b6b277b0275105e9aa0ab39e9
Nov 02 01:46:58.956740 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:58.956714    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"]
Nov 02 01:46:59.131103 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:59.131063    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" event={"ID":"169d2fdb-6588-4d7e-9892-2d12c0e017b6","Type":"ContainerStarted","Data":"9e68f4c82721f73f52a8e70236c5f8b47e05b8d4d8e2f7ed394fa6ef869543ff"}
Nov 02 01:46:59.131271 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:59.131112    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" event={"ID":"169d2fdb-6588-4d7e-9892-2d12c0e017b6","Type":"ContainerStarted","Data":"e3e4fac4aa265480c4e85b8806d366073b10965b6b277b0275105e9aa0ab39e9"}
Nov 02 01:46:59.250859 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:59.250758    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podStartSLOduration=1.250743291 podStartE2EDuration="1.250743291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:46:59.247744779 +0000 UTC m=+1284.151426892" watchObservedRunningTime="2026-11-02 01:46:59.250743291 +0000 UTC m=+1284.154425404"
Nov 02 01:46:59.440453 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:59.440415    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:46:59.693004 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:59.692975    5639 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 01:46:59.693162 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:46:59.693020    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx" podUID="cd05aad1-3b5f-43bd-988e-fce23728a36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 01:47:00.692681 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:00.692647    5639 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 01:47:00.693066 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:00.692714    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx" podUID="cd05aad1-3b5f-43bd-988e-fce23728a36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 01:47:01.089993 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:01.089961    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c52101d3-vnp2p"]
Nov 02 01:47:01.094219 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:01.094202    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52101d3-vnp2p"
Nov 02 01:47:01.196506 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:01.196466    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:47:01.196724 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:01.196514    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:47:01.196724 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:01.196568    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:47:01.196868 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:01.196759    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:47:01.200689 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:01.200667    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c52101d3-vnp2p"]
Nov 02 01:47:01.202390 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:01.202369    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvmvs\" (UniqueName: \"kubernetes.io/projected/66dd198e-ff0a-4b6b-9011-32adb132e77c-kube-api-access-rvmvs\") pod \"maas-api-revoke-sub-c52101d3-vnp2p\" (UID: \"66dd198e-ff0a-4b6b-9011-32adb132e77c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52101d3-vnp2p"
Nov 02 01:47:01.202457 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:01.202439    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66dd198e-ff0a-4b6b-9011-32adb132e77c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c52101d3-vnp2p\" (UID: \"66dd198e-ff0a-4b6b-9011-32adb132e77c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52101d3-vnp2p"
Nov 02 01:47:01.202602 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:01.202583    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66dd198e-ff0a-4b6b-9011-32adb132e77c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c52101d3-vnp2p\" (UID: \"66dd198e-ff0a-4b6b-9011-32adb132e77c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52101d3-vnp2p"
Nov 02 01:47:01.304163 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:01.304130    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvmvs\" (UniqueName: \"kubernetes.io/projected/66dd198e-ff0a-4b6b-9011-32adb132e77c-kube-api-access-rvmvs\") pod \"maas-api-revoke-sub-c52101d3-vnp2p\" (UID: \"66dd198e-ff0a-4b6b-9011-32adb132e77c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52101d3-vnp2p"
Nov 02 01:47:01.304372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:01.304211    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66dd198e-ff0a-4b6b-9011-32adb132e77c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c52101d3-vnp2p\" (UID: \"66dd198e-ff0a-4b6b-9011-32adb132e77c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52101d3-vnp2p"
Nov 02 01:47:01.304372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:01.304343    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66dd198e-ff0a-4b6b-9011-32adb132e77c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c52101d3-vnp2p\" (UID: \"66dd198e-ff0a-4b6b-9011-32adb132e77c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52101d3-vnp2p"
Nov 02 01:47:01.304929 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:01.304904    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66dd198e-ff0a-4b6b-9011-32adb132e77c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c52101d3-vnp2p\" (UID: \"66dd198e-ff0a-4b6b-9011-32adb132e77c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52101d3-vnp2p"
Nov 02 01:47:01.306808 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:01.306776    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66dd198e-ff0a-4b6b-9011-32adb132e77c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c52101d3-vnp2p\" (UID: \"66dd198e-ff0a-4b6b-9011-32adb132e77c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52101d3-vnp2p"
Nov 02 01:47:01.384071 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:01.383991    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvmvs\" (UniqueName: \"kubernetes.io/projected/66dd198e-ff0a-4b6b-9011-32adb132e77c-kube-api-access-rvmvs\") pod \"maas-api-revoke-sub-c52101d3-vnp2p\" (UID: \"66dd198e-ff0a-4b6b-9011-32adb132e77c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52101d3-vnp2p"
Nov 02 01:47:01.408779 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:01.408733    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52101d3-vnp2p"
Nov 02 01:47:01.693312 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:01.693227    5639 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 01:47:01.693669 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:01.693309    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx" podUID="cd05aad1-3b5f-43bd-988e-fce23728a36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 01:47:01.825145 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:01.825111    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c52101d3-vnp2p"]
Nov 02 01:47:01.825524 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:47:01.825501    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66dd198e_ff0a_4b6b_9011_32adb132e77c.slice/crio-b665a538e92c764be1e28de339bb963f0dfb03d34520b6dce3c5854cd148dcce WatchSource:0}: Error finding container b665a538e92c764be1e28de339bb963f0dfb03d34520b6dce3c5854cd148dcce: Status 404 returned error can't find the container with id b665a538e92c764be1e28de339bb963f0dfb03d34520b6dce3c5854cd148dcce
Nov 02 01:47:02.153610 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:02.153577    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52101d3-vnp2p" event={"ID":"66dd198e-ff0a-4b6b-9011-32adb132e77c","Type":"ContainerStarted","Data":"b665a538e92c764be1e28de339bb963f0dfb03d34520b6dce3c5854cd148dcce"}
Nov 02 01:47:02.692839 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:02.692810    5639 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 01:47:02.693021 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:02.692858    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx" podUID="cd05aad1-3b5f-43bd-988e-fce23728a36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 01:47:03.163094 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:03.163057    5639 generic.go:356] "Generic (PLEG): container finished" podID="66dd198e-ff0a-4b6b-9011-32adb132e77c" containerID="10f596c2764d773d7680158b311e4b34000375428da7de77d4af2380065ad524" exitCode=0
Nov 02 01:47:03.163659 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:03.163145    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52101d3-vnp2p" event={"ID":"66dd198e-ff0a-4b6b-9011-32adb132e77c","Type":"ContainerDied","Data":"10f596c2764d773d7680158b311e4b34000375428da7de77d4af2380065ad524"}
Nov 02 01:47:03.697190 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:03.697159    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:47:03.697649 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:03.697629    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:47:03.698460 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:03.698440    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:47:04.311140 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:04.311117    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52101d3-vnp2p"
Nov 02 01:47:04.338102 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:04.338071    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66dd198e-ff0a-4b6b-9011-32adb132e77c-kube-api-access-rvmvs\" (UniqueName: \"kubernetes.io/projected/66dd198e-ff0a-4b6b-9011-32adb132e77c-kube-api-access-rvmvs\") pod \"66dd198e-ff0a-4b6b-9011-32adb132e77c\" (UID: \"66dd198e-ff0a-4b6b-9011-32adb132e77c\") "
Nov 02 01:47:04.338291 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:04.338223    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66dd198e-ff0a-4b6b-9011-32adb132e77c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66dd198e-ff0a-4b6b-9011-32adb132e77c-maas-api-openshift-ca\") pod \"66dd198e-ff0a-4b6b-9011-32adb132e77c\" (UID: \"66dd198e-ff0a-4b6b-9011-32adb132e77c\") "
Nov 02 01:47:04.338376 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:04.338310    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66dd198e-ff0a-4b6b-9011-32adb132e77c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66dd198e-ff0a-4b6b-9011-32adb132e77c-maas-api-xks-ca\") pod \"66dd198e-ff0a-4b6b-9011-32adb132e77c\" (UID: \"66dd198e-ff0a-4b6b-9011-32adb132e77c\") "
Nov 02 01:47:04.338566 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:04.338540    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66dd198e-ff0a-4b6b-9011-32adb132e77c-maas-api-openshift-ca" pod "66dd198e-ff0a-4b6b-9011-32adb132e77c" (UID: "66dd198e-ff0a-4b6b-9011-32adb132e77c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:47:04.338731 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:04.338711    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66dd198e-ff0a-4b6b-9011-32adb132e77c-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:04.340451 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:04.340426    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66dd198e-ff0a-4b6b-9011-32adb132e77c-maas-api-xks-ca" pod "66dd198e-ff0a-4b6b-9011-32adb132e77c" (UID: "66dd198e-ff0a-4b6b-9011-32adb132e77c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:47:04.340451 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:04.340445    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66dd198e-ff0a-4b6b-9011-32adb132e77c-kube-api-access-rvmvs" pod "66dd198e-ff0a-4b6b-9011-32adb132e77c" (UID: "66dd198e-ff0a-4b6b-9011-32adb132e77c"). InnerVolumeSpecName "kube-api-access-rvmvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:47:04.442663 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:04.442371    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66dd198e-ff0a-4b6b-9011-32adb132e77c-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:04.442663 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:04.442606    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvmvs\" (UniqueName: \"kubernetes.io/projected/66dd198e-ff0a-4b6b-9011-32adb132e77c-kube-api-access-rvmvs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:05.076857 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:05.076824    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04784feb-vctkw"]
Nov 02 01:47:05.080127 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:05.080106    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04784feb-vctkw"]
Nov 02 01:47:05.180749 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:05.180715    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52101d3-vnp2p" event={"ID":"66dd198e-ff0a-4b6b-9011-32adb132e77c","Type":"ContainerDied","Data":"b665a538e92c764be1e28de339bb963f0dfb03d34520b6dce3c5854cd148dcce"}
Nov 02 01:47:05.180926 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:05.180794    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b665a538e92c764be1e28de339bb963f0dfb03d34520b6dce3c5854cd148dcce"
Nov 02 01:47:05.180926 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:05.180729    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52101d3-vnp2p"
Nov 02 01:47:05.269583 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:05.269548    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56fdfb40-2f99-4420-a0c3-179f8a740250" path="/var/lib/kubelet/pods/56fdfb40-2f99-4420-a0c3-179f8a740250/volumes"
Nov 02 01:47:08.435526 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:08.435489    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:47:08.449370 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:08.449344    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:47:09.220004 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:09.219974    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:47:10.212070 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.212040    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"]
Nov 02 01:47:10.213129 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.213110    5639 cpu_manager.go:414] "containerMap: removing container" podUID="66dd198e-ff0a-4b6b-9011-32adb132e77c" containerName="revoke-keys"
Nov 02 01:47:10.213192 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.213131    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66dd198e-ff0a-4b6b-9011-32adb132e77c" containerName="revoke-keys"
Nov 02 01:47:10.213230 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.213222    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="66dd198e-ff0a-4b6b-9011-32adb132e77c" containerName="revoke-keys"
Nov 02 01:47:10.218340 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.218318    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.318414 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.318380    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.318596 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.318486    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9984c750-63ba-4996-a4b1-44549030fe24-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.318659 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.318596    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9984c750-63ba-4996-a4b1-44549030fe24-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.318784 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.318758    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.318961 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.318940    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9984c750-63ba-4996-a4b1-44549030fe24-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.319026 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.319007    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.319062 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.319052    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl2v9\" (UniqueName: \"kubernetes.io/projected/9984c750-63ba-4996-a4b1-44549030fe24-kube-api-access-vl2v9\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.319100 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.319080    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.319134 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.319101    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.325545 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.325515    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"]
Nov 02 01:47:10.420796 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.420762    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl2v9\" (UniqueName: \"kubernetes.io/projected/9984c750-63ba-4996-a4b1-44549030fe24-kube-api-access-vl2v9\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.420796 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.420797    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.421073 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.420814    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.421073 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.420904    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.421073 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.420949    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9984c750-63ba-4996-a4b1-44549030fe24-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.421073 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.421000    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9984c750-63ba-4996-a4b1-44549030fe24-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.421073 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.421069    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.421390 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.421096    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9984c750-63ba-4996-a4b1-44549030fe24-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.421390 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.421165    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.421390 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.421317    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.421554 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.421534    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.421625 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.421579    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.421625 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.421605    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.421761 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.421745    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9984c750-63ba-4996-a4b1-44549030fe24-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.423377 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.423356    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.423689 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.423668    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9984c750-63ba-4996-a4b1-44549030fe24-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.481588 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.481506    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9984c750-63ba-4996-a4b1-44549030fe24-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.481758 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.481720    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl2v9\" (UniqueName: \"kubernetes.io/projected/9984c750-63ba-4996-a4b1-44549030fe24-kube-api-access-vl2v9\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.533646 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.533615    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:10.965736 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.965706    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"]
Nov 02 01:47:10.966905 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:47:10.966868    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9984c750_63ba_4996_a4b1_44549030fe24.slice/crio-e86c33e8f80d06c8250ce38ebf33c6a4b63c786bcdb48a637855a38191a57d36 WatchSource:0}: Error finding container e86c33e8f80d06c8250ce38ebf33c6a4b63c786bcdb48a637855a38191a57d36: Status 404 returned error can't find the container with id e86c33e8f80d06c8250ce38ebf33c6a4b63c786bcdb48a637855a38191a57d36
Nov 02 01:47:10.969336 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.969260    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:47:10.969420 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.969372    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:47:10.969420 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:10.969402    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:47:11.049179 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:11.049140    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a8a1d0a-qtms2"]
Nov 02 01:47:11.053265 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:11.053241    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a8a1d0a-qtms2"]
Nov 02 01:47:11.219327 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:11.219207    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8dc7940-e89b-4d9c-ab81-2c66fe140456" path="/var/lib/kubelet/pods/a8dc7940-e89b-4d9c-ab81-2c66fe140456/volumes"
Nov 02 01:47:11.229133 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:11.229100    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5" event={"ID":"9984c750-63ba-4996-a4b1-44549030fe24","Type":"ContainerStarted","Data":"f62dd4fc2e17f5b9b6c749612001a52f659c6a2f5f9fc324725bdab67d48a4e6"}
Nov 02 01:47:11.229371 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:11.229166    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5" event={"ID":"9984c750-63ba-4996-a4b1-44549030fe24","Type":"ContainerStarted","Data":"e86c33e8f80d06c8250ce38ebf33c6a4b63c786bcdb48a637855a38191a57d36"}
Nov 02 01:47:11.354794 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:11.354741    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5" podStartSLOduration=1.354727026 podStartE2EDuration="1.354727026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:47:11.353809717 +0000 UTC m=+1296.257491857" watchObservedRunningTime="2026-11-02 01:47:11.354727026 +0000 UTC m=+1296.258409139"
Nov 02 01:47:11.534770 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:11.534691    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:12.535856 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:12.535817    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 01:47:12.536334 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:12.535891    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5" podUID="9984c750-63ba-4996-a4b1-44549030fe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 01:47:13.534797 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:13.534758    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 01:47:13.534992 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:13.534824    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5" podUID="9984c750-63ba-4996-a4b1-44549030fe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 01:47:13.733613 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:13.733582    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74e25620-mpqgc"]
Nov 02 01:47:13.738084 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:13.738064    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e25620-mpqgc"
Nov 02 01:47:13.795228 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:13.795145    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74e25620-mpqgc"]
Nov 02 01:47:13.839942 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:13.839912    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:47:13.840172 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:13.840148    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:47:13.840238 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:13.840164    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:47:13.842994 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:13.842972    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:47:13.885070 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:13.881188    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tmhk\" (UniqueName: \"kubernetes.io/projected/ca4c83be-268c-4163-8709-61c64ad2e750-kube-api-access-9tmhk\") pod \"maas-api-revoke-sub-74e25620-mpqgc\" (UID: \"ca4c83be-268c-4163-8709-61c64ad2e750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e25620-mpqgc"
Nov 02 01:47:13.885070 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:13.881245    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca4c83be-268c-4163-8709-61c64ad2e750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74e25620-mpqgc\" (UID: \"ca4c83be-268c-4163-8709-61c64ad2e750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e25620-mpqgc"
Nov 02 01:47:13.885070 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:13.881460    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca4c83be-268c-4163-8709-61c64ad2e750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74e25620-mpqgc\" (UID: \"ca4c83be-268c-4163-8709-61c64ad2e750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e25620-mpqgc"
Nov 02 01:47:13.997302 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:13.991134    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca4c83be-268c-4163-8709-61c64ad2e750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74e25620-mpqgc\" (UID: \"ca4c83be-268c-4163-8709-61c64ad2e750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e25620-mpqgc"
Nov 02 01:47:13.997302 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:13.991333    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tmhk\" (UniqueName: \"kubernetes.io/projected/ca4c83be-268c-4163-8709-61c64ad2e750-kube-api-access-9tmhk\") pod \"maas-api-revoke-sub-74e25620-mpqgc\" (UID: \"ca4c83be-268c-4163-8709-61c64ad2e750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e25620-mpqgc"
Nov 02 01:47:13.997302 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:13.991364    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca4c83be-268c-4163-8709-61c64ad2e750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74e25620-mpqgc\" (UID: \"ca4c83be-268c-4163-8709-61c64ad2e750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e25620-mpqgc"
Nov 02 01:47:14.026323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:14.023486    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca4c83be-268c-4163-8709-61c64ad2e750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74e25620-mpqgc\" (UID: \"ca4c83be-268c-4163-8709-61c64ad2e750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e25620-mpqgc"
Nov 02 01:47:14.030305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:14.029064    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca4c83be-268c-4163-8709-61c64ad2e750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74e25620-mpqgc\" (UID: \"ca4c83be-268c-4163-8709-61c64ad2e750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e25620-mpqgc"
Nov 02 01:47:14.119307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:14.119055    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tmhk\" (UniqueName: \"kubernetes.io/projected/ca4c83be-268c-4163-8709-61c64ad2e750-kube-api-access-9tmhk\") pod \"maas-api-revoke-sub-74e25620-mpqgc\" (UID: \"ca4c83be-268c-4163-8709-61c64ad2e750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e25620-mpqgc"
Nov 02 01:47:14.359313 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:14.358499    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e25620-mpqgc"
Nov 02 01:47:14.537508 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:14.537472    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 01:47:14.537707 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:14.537608    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5" podUID="9984c750-63ba-4996-a4b1-44549030fe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 01:47:14.930426 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:14.930384    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:15.250396 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:15.244377    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:15.507539 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:15.415335    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:15.554047 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:15.552754    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 01:47:15.554047 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:15.552817    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5" podUID="9984c750-63ba-4996-a4b1-44549030fe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 01:47:15.637307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:15.633405    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:15.637307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:15.634166    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74e25620-mpqgc"]
Nov 02 01:47:15.669901 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:15.662514    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:47:15.959624 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:15.940057    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:16.000309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:15.987389    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Nov 02 01:47:16.000309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:15.987759    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:47:16.000309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:15.988003    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:16.230146 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:16.230065    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e07cc748-4qgfl"]
Nov 02 01:47:16.250306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:16.244631    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:16.250306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:16.245875    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:47:16.277759 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:16.277636    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:16.454320 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:16.426402    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:16.454320 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:16.426844    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:47:16.612909 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:16.612871    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e25620-mpqgc" event={"ID":"ca4c83be-268c-4163-8709-61c64ad2e750","Type":"ContainerStarted","Data":"76d8182365ec756f86e937cbf206d51ac8d516801a586c2f8b8674c9562c1550"}
Nov 02 01:47:16.644999 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:16.644957    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:47:16.781476 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:16.659643    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:47:16.781476 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:16.769430    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:47:16.785305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:16.662264    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Nov 02 01:47:16.785305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:16.688471    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:16.785535 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:16.688578    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 01:47:16.785535 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:16.785467    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5" podUID="9984c750-63ba-4996-a4b1-44549030fe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 01:47:16.785642 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:16.785604    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:47:16.822664 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:16.822616    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e07cc748-4qgfl"]
Nov 02 01:47:16.957401 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:16.956529    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 02 01:47:17.022369 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:17.019413    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:17.022369 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:17.019985    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:47:17.314421 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:17.314371    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:17.325110 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:17.323440    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 02 01:47:17.466304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:17.449713    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:47:17.466304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:17.451372    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:47:17.466304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:17.451413    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2" podUID="c4dec6db-d72f-45ce-802c-f54bb7f57e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:17.559617 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:17.559579    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03e61650-f799-4c4d-8ca6-ef03609fd624" path="/var/lib/kubelet/pods/03e61650-f799-4c4d-8ca6-ef03609fd624/volumes"
Nov 02 01:47:17.560354 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:17.560330    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:47:17.600365 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:17.575579    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:47:17.611177 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:17.610469    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 01:47:17.652442 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:17.652394    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5" podUID="9984c750-63ba-4996-a4b1-44549030fe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 01:47:17.829307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:17.777526    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:17.829307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:17.782344    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:17.829307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:17.790342    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:47:17.829307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:17.792103    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:47:17.873372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:17.866336    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:47:17.964509 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:17.964454    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:17.988051 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:17.988011    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:18.024630 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:18.024570    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Nov 02 01:47:18.048336 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:18.047371    5639 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:47:18.048336 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:18.047433    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp" podUID="02d298db-5bc9-469a-af6b-ba7fa45f8602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:18.150672 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:18.149086    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:47:18.150672 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:18.150341    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:47:18.325538 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:18.325331    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:18.331411 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:18.331164    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:47:18.431959 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:18.431878    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:47:18.434634 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:18.434611    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:47:18.443149 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:18.443126    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:47:18.535800 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:18.535549    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 01:47:18.536805 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:18.536764    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5" podUID="9984c750-63ba-4996-a4b1-44549030fe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 01:47:18.926843 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:18.926785    5639 generic.go:356] "Generic (PLEG): container finished" podID="ca4c83be-268c-4163-8709-61c64ad2e750" containerID="5ce026485571e5be2f3db4ef87bed6fecd2af03cf296d75cf91c439e4c742916" exitCode=0
Nov 02 01:47:18.927042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:18.926990    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e25620-mpqgc" event={"ID":"ca4c83be-268c-4163-8709-61c64ad2e750","Type":"ContainerDied","Data":"5ce026485571e5be2f3db4ef87bed6fecd2af03cf296d75cf91c439e4c742916"}
Nov 02 01:47:19.266363 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:19.266325    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-564a8351-h2snf"]
Nov 02 01:47:19.271694 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:19.271668    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a8351-h2snf"
Nov 02 01:47:19.328982 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:19.328939    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-564a8351-h2snf"]
Nov 02 01:47:19.361120 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:19.359820    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c229fa69-e9fd-4d5c-bfca-979c4602a398-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-564a8351-h2snf\" (UID: \"c229fa69-e9fd-4d5c-bfca-979c4602a398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a8351-h2snf"
Nov 02 01:47:19.361120 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:19.359881    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c229fa69-e9fd-4d5c-bfca-979c4602a398-maas-api-xks-ca\") pod \"maas-api-revoke-sub-564a8351-h2snf\" (UID: \"c229fa69-e9fd-4d5c-bfca-979c4602a398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a8351-h2snf"
Nov 02 01:47:19.361120 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:19.359936    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w882\" (UniqueName: \"kubernetes.io/projected/c229fa69-e9fd-4d5c-bfca-979c4602a398-kube-api-access-6w882\") pod \"maas-api-revoke-sub-564a8351-h2snf\" (UID: \"c229fa69-e9fd-4d5c-bfca-979c4602a398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a8351-h2snf"
Nov 02 01:47:19.461067 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:19.461014    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w882\" (UniqueName: \"kubernetes.io/projected/c229fa69-e9fd-4d5c-bfca-979c4602a398-kube-api-access-6w882\") pod \"maas-api-revoke-sub-564a8351-h2snf\" (UID: \"c229fa69-e9fd-4d5c-bfca-979c4602a398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a8351-h2snf"
Nov 02 01:47:19.461418 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:19.461393    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c229fa69-e9fd-4d5c-bfca-979c4602a398-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-564a8351-h2snf\" (UID: \"c229fa69-e9fd-4d5c-bfca-979c4602a398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a8351-h2snf"
Nov 02 01:47:19.461584 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:19.461436    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c229fa69-e9fd-4d5c-bfca-979c4602a398-maas-api-xks-ca\") pod \"maas-api-revoke-sub-564a8351-h2snf\" (UID: \"c229fa69-e9fd-4d5c-bfca-979c4602a398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a8351-h2snf"
Nov 02 01:47:19.462161 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:19.462134    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c229fa69-e9fd-4d5c-bfca-979c4602a398-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-564a8351-h2snf\" (UID: \"c229fa69-e9fd-4d5c-bfca-979c4602a398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a8351-h2snf"
Nov 02 01:47:19.464007 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:19.463987    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c229fa69-e9fd-4d5c-bfca-979c4602a398-maas-api-xks-ca\") pod \"maas-api-revoke-sub-564a8351-h2snf\" (UID: \"c229fa69-e9fd-4d5c-bfca-979c4602a398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a8351-h2snf"
Nov 02 01:47:19.522366 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:19.522267    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w882\" (UniqueName: \"kubernetes.io/projected/c229fa69-e9fd-4d5c-bfca-979c4602a398-kube-api-access-6w882\") pod \"maas-api-revoke-sub-564a8351-h2snf\" (UID: \"c229fa69-e9fd-4d5c-bfca-979c4602a398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a8351-h2snf"
Nov 02 01:47:19.534467 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:19.534439    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 01:47:19.534624 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:19.534525    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5" podUID="9984c750-63ba-4996-a4b1-44549030fe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 01:47:19.589559 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:19.589518    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a8351-h2snf"
Nov 02 01:47:19.988366 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:19.987908    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-564a8351-h2snf"]
Nov 02 01:47:20.117197 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:20.117172    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e25620-mpqgc"
Nov 02 01:47:20.185558 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:20.185513    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca4c83be-268c-4163-8709-61c64ad2e750-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca4c83be-268c-4163-8709-61c64ad2e750-maas-api-openshift-ca\") pod \"ca4c83be-268c-4163-8709-61c64ad2e750\" (UID: \"ca4c83be-268c-4163-8709-61c64ad2e750\") "
Nov 02 01:47:20.185709 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:20.185628    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca4c83be-268c-4163-8709-61c64ad2e750-kube-api-access-9tmhk\" (UniqueName: \"kubernetes.io/projected/ca4c83be-268c-4163-8709-61c64ad2e750-kube-api-access-9tmhk\") pod \"ca4c83be-268c-4163-8709-61c64ad2e750\" (UID: \"ca4c83be-268c-4163-8709-61c64ad2e750\") "
Nov 02 01:47:20.185949 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:20.185912    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca4c83be-268c-4163-8709-61c64ad2e750-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca4c83be-268c-4163-8709-61c64ad2e750-maas-api-xks-ca\") pod \"ca4c83be-268c-4163-8709-61c64ad2e750\" (UID: \"ca4c83be-268c-4163-8709-61c64ad2e750\") "
Nov 02 01:47:20.186066 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:20.185942    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca4c83be-268c-4163-8709-61c64ad2e750-maas-api-openshift-ca" pod "ca4c83be-268c-4163-8709-61c64ad2e750" (UID: "ca4c83be-268c-4163-8709-61c64ad2e750"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:47:20.186583 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:20.186558    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca4c83be-268c-4163-8709-61c64ad2e750-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:20.188480 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:20.188424    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca4c83be-268c-4163-8709-61c64ad2e750-maas-api-xks-ca" pod "ca4c83be-268c-4163-8709-61c64ad2e750" (UID: "ca4c83be-268c-4163-8709-61c64ad2e750"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:47:20.189262 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:20.189239    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca4c83be-268c-4163-8709-61c64ad2e750-kube-api-access-9tmhk" pod "ca4c83be-268c-4163-8709-61c64ad2e750" (UID: "ca4c83be-268c-4163-8709-61c64ad2e750"). InnerVolumeSpecName "kube-api-access-9tmhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:47:20.288238 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:20.288204    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca4c83be-268c-4163-8709-61c64ad2e750-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:20.288238 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:20.288241    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tmhk\" (UniqueName: \"kubernetes.io/projected/ca4c83be-268c-4163-8709-61c64ad2e750-kube-api-access-9tmhk\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:20.613347 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:20.613271    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:20.614343 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:20.614317    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:20.615108 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:20.615083    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:47:20.947699 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:20.947660    5639 generic.go:356] "Generic (PLEG): container finished" podID="c229fa69-e9fd-4d5c-bfca-979c4602a398" containerID="1d0feb417424c7c7c318cf627dfad88b531bb3f1e0b8f645ec90a8654caedcf0" exitCode=0
Nov 02 01:47:20.947881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:20.947746    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a8351-h2snf" event={"ID":"c229fa69-e9fd-4d5c-bfca-979c4602a398","Type":"ContainerDied","Data":"1d0feb417424c7c7c318cf627dfad88b531bb3f1e0b8f645ec90a8654caedcf0"}
Nov 02 01:47:20.947881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:20.947829    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a8351-h2snf" event={"ID":"c229fa69-e9fd-4d5c-bfca-979c4602a398","Type":"ContainerStarted","Data":"5a5cf6decea32074efc61c69b08e81545eb07529668caa91ac6492a13a025790"}
Nov 02 01:47:20.949550 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:20.949529    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e25620-mpqgc"
Nov 02 01:47:20.949673 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:20.949581    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e25620-mpqgc" event={"ID":"ca4c83be-268c-4163-8709-61c64ad2e750","Type":"ContainerDied","Data":"76d8182365ec756f86e937cbf206d51ac8d516801a586c2f8b8674c9562c1550"}
Nov 02 01:47:20.949673 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:20.949628    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76d8182365ec756f86e937cbf206d51ac8d516801a586c2f8b8674c9562c1550"
Nov 02 01:47:22.104242 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:22.104217    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a8351-h2snf"
Nov 02 01:47:22.216873 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:22.216769    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c229fa69-e9fd-4d5c-bfca-979c4602a398-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c229fa69-e9fd-4d5c-bfca-979c4602a398-maas-api-xks-ca\") pod \"c229fa69-e9fd-4d5c-bfca-979c4602a398\" (UID: \"c229fa69-e9fd-4d5c-bfca-979c4602a398\") "
Nov 02 01:47:22.217047 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:22.216900    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c229fa69-e9fd-4d5c-bfca-979c4602a398-kube-api-access-6w882\" (UniqueName: \"kubernetes.io/projected/c229fa69-e9fd-4d5c-bfca-979c4602a398-kube-api-access-6w882\") pod \"c229fa69-e9fd-4d5c-bfca-979c4602a398\" (UID: \"c229fa69-e9fd-4d5c-bfca-979c4602a398\") "
Nov 02 01:47:22.217047 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:22.216934    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c229fa69-e9fd-4d5c-bfca-979c4602a398-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c229fa69-e9fd-4d5c-bfca-979c4602a398-maas-api-openshift-ca\") pod \"c229fa69-e9fd-4d5c-bfca-979c4602a398\" (UID: \"c229fa69-e9fd-4d5c-bfca-979c4602a398\") "
Nov 02 01:47:22.217387 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:22.217348    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c229fa69-e9fd-4d5c-bfca-979c4602a398-maas-api-openshift-ca" pod "c229fa69-e9fd-4d5c-bfca-979c4602a398" (UID: "c229fa69-e9fd-4d5c-bfca-979c4602a398"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:47:22.217639 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:22.217606    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c229fa69-e9fd-4d5c-bfca-979c4602a398-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:22.218995 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:22.218972    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c229fa69-e9fd-4d5c-bfca-979c4602a398-maas-api-xks-ca" pod "c229fa69-e9fd-4d5c-bfca-979c4602a398" (UID: "c229fa69-e9fd-4d5c-bfca-979c4602a398"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:47:22.219148 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:22.219127    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c229fa69-e9fd-4d5c-bfca-979c4602a398-kube-api-access-6w882" pod "c229fa69-e9fd-4d5c-bfca-979c4602a398" (UID: "c229fa69-e9fd-4d5c-bfca-979c4602a398"). InnerVolumeSpecName "kube-api-access-6w882". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:47:22.319223 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:22.319186    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c229fa69-e9fd-4d5c-bfca-979c4602a398-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:22.319223 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:22.319219    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w882\" (UniqueName: \"kubernetes.io/projected/c229fa69-e9fd-4d5c-bfca-979c4602a398-kube-api-access-6w882\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:22.970696 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:22.970542    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a8351-h2snf"
Nov 02 01:47:22.970937 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:22.970544    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a8351-h2snf" event={"ID":"c229fa69-e9fd-4d5c-bfca-979c4602a398","Type":"ContainerDied","Data":"5a5cf6decea32074efc61c69b08e81545eb07529668caa91ac6492a13a025790"}
Nov 02 01:47:22.970937 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:22.970790    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a5cf6decea32074efc61c69b08e81545eb07529668caa91ac6492a13a025790"
Nov 02 01:47:24.309822 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.309787    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"]
Nov 02 01:47:24.310432 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.310417    5639 cpu_manager.go:414] "containerMap: removing container" podUID="ca4c83be-268c-4163-8709-61c64ad2e750" containerName="revoke-keys"
Nov 02 01:47:24.310482 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.310433    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca4c83be-268c-4163-8709-61c64ad2e750" containerName="revoke-keys"
Nov 02 01:47:24.310482 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.310449    5639 cpu_manager.go:414] "containerMap: removing container" podUID="c229fa69-e9fd-4d5c-bfca-979c4602a398" containerName="revoke-keys"
Nov 02 01:47:24.310482 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.310454    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c229fa69-e9fd-4d5c-bfca-979c4602a398" containerName="revoke-keys"
Nov 02 01:47:24.310573 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.310545    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca4c83be-268c-4163-8709-61c64ad2e750" containerName="revoke-keys"
Nov 02 01:47:24.310573 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.310553    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="c229fa69-e9fd-4d5c-bfca-979c4602a398" containerName="revoke-keys"
Nov 02 01:47:24.314097 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.314078    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.422233 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.422201    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"]
Nov 02 01:47:24.447084 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.447053    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.447300 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.447101    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.447300 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.447134    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.447446 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.447346    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.447446 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.447388    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.447446 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.447433    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tcjj\" (UniqueName: \"kubernetes.io/projected/82ab4f70-ac0f-42f7-8571-93b7ef884c77-kube-api-access-2tcjj\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.447599 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.447470    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.447599 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.447516    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.447725 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.447706    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.549144 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.549110    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.549144 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.549146    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.549414 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.549175    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tcjj\" (UniqueName: \"kubernetes.io/projected/82ab4f70-ac0f-42f7-8571-93b7ef884c77-kube-api-access-2tcjj\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.549414 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.549319    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.549414 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.549381    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.549568 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.549554    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.549568 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.549556    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.549654 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.549593    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.549654 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.549622    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.549730 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.549702    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.549848 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.549818    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.549981 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.549858    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.549981 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.549926    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.550367 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.550351    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.552359 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.552332    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.552545 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.552526    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.616101 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.616069    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.616377 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.616247    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tcjj\" (UniqueName: \"kubernetes.io/projected/82ab4f70-ac0f-42f7-8571-93b7ef884c77-kube-api-access-2tcjj\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:24.627972 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:24.627945    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:25.066828 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:25.066794    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"]
Nov 02 01:47:25.067738 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:47:25.067709    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ab4f70_ac0f_42f7_8571_93b7ef884c77.slice/crio-853e4d4c1b9023cb1f961c9e508c55621cd288422f31b2c9622af0116995ad1e WatchSource:0}: Error finding container 853e4d4c1b9023cb1f961c9e508c55621cd288422f31b2c9622af0116995ad1e: Status 404 returned error can't find the container with id 853e4d4c1b9023cb1f961c9e508c55621cd288422f31b2c9622af0116995ad1e
Nov 02 01:47:25.070567 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:25.070535    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:47:25.070648 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:25.070598    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:47:25.070648 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:25.070630    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:47:25.994731 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:25.994692    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52" event={"ID":"82ab4f70-ac0f-42f7-8571-93b7ef884c77","Type":"ContainerStarted","Data":"54f4ac87f7e1b5bb9553cfc5f69715dad39b03e6cf1570ac0b30b8513dd85cab"}
Nov 02 01:47:25.995150 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:25.994760    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52" event={"ID":"82ab4f70-ac0f-42f7-8571-93b7ef884c77","Type":"ContainerStarted","Data":"853e4d4c1b9023cb1f961c9e508c55621cd288422f31b2c9622af0116995ad1e"}
Nov 02 01:47:26.187649 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:26.187580    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52" podStartSLOduration=2.187563205 podStartE2EDuration="2.187563205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:47:26.165714256 +0000 UTC m=+1311.069396371" watchObservedRunningTime="2026-11-02 01:47:26.187563205 +0000 UTC m=+1311.091245319"
Nov 02 01:47:26.629041 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:26.629008    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:26.630847 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:26.630815    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 01:47:26.630987 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:26.630865    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52" podUID="82ab4f70-ac0f-42f7-8571-93b7ef884c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 01:47:27.420967 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:47:27.420928    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:47:27.629295 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:27.629249    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 01:47:27.629467 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:27.629346    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52" podUID="82ab4f70-ac0f-42f7-8571-93b7ef884c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 01:47:28.385169 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.385132    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a237e48-44z8t"]
Nov 02 01:47:28.389782 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.389757    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a237e48-44z8t"
Nov 02 01:47:28.423244 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.423206    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a237e48-44z8t"]
Nov 02 01:47:28.430821 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.430787    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0096d1bd-5a8d-44fd-be18-d3b269ea8107-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a237e48-44z8t\" (UID: \"0096d1bd-5a8d-44fd-be18-d3b269ea8107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a237e48-44z8t"
Nov 02 01:47:28.430995 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.430856    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0096d1bd-5a8d-44fd-be18-d3b269ea8107-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a237e48-44z8t\" (UID: \"0096d1bd-5a8d-44fd-be18-d3b269ea8107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a237e48-44z8t"
Nov 02 01:47:28.431061 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.431029    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w96g5\" (UniqueName: \"kubernetes.io/projected/0096d1bd-5a8d-44fd-be18-d3b269ea8107-kube-api-access-w96g5\") pod \"maas-api-revoke-sub-1a237e48-44z8t\" (UID: \"0096d1bd-5a8d-44fd-be18-d3b269ea8107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a237e48-44z8t"
Nov 02 01:47:28.486737 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.486704    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:47:28.487158 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.487140    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:47:28.487387 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.487375    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:47:28.487542 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.487530    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:47:28.496704 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.496678    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bd4274e-nnf2h"]
Nov 02 01:47:28.500886 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.500869    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd4274e-nnf2h"
Nov 02 01:47:28.532371 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.532343    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w96g5\" (UniqueName: \"kubernetes.io/projected/0096d1bd-5a8d-44fd-be18-d3b269ea8107-kube-api-access-w96g5\") pod \"maas-api-revoke-sub-1a237e48-44z8t\" (UID: \"0096d1bd-5a8d-44fd-be18-d3b269ea8107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a237e48-44z8t"
Nov 02 01:47:28.532534 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.532449    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2gtk\" (UniqueName: \"kubernetes.io/projected/a30fb1a7-0467-4adc-928f-6a4a2eadb714-kube-api-access-k2gtk\") pod \"maas-api-revoke-sub-8bd4274e-nnf2h\" (UID: \"a30fb1a7-0467-4adc-928f-6a4a2eadb714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd4274e-nnf2h"
Nov 02 01:47:28.532534 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.532503    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a30fb1a7-0467-4adc-928f-6a4a2eadb714-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bd4274e-nnf2h\" (UID: \"a30fb1a7-0467-4adc-928f-6a4a2eadb714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd4274e-nnf2h"
Nov 02 01:47:28.532636 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.532623    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0096d1bd-5a8d-44fd-be18-d3b269ea8107-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a237e48-44z8t\" (UID: \"0096d1bd-5a8d-44fd-be18-d3b269ea8107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a237e48-44z8t"
Nov 02 01:47:28.532673 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.532651    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a30fb1a7-0467-4adc-928f-6a4a2eadb714-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bd4274e-nnf2h\" (UID: \"a30fb1a7-0467-4adc-928f-6a4a2eadb714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd4274e-nnf2h"
Nov 02 01:47:28.532703 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.532677    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0096d1bd-5a8d-44fd-be18-d3b269ea8107-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a237e48-44z8t\" (UID: \"0096d1bd-5a8d-44fd-be18-d3b269ea8107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a237e48-44z8t"
Nov 02 01:47:28.533315 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.533293    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0096d1bd-5a8d-44fd-be18-d3b269ea8107-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a237e48-44z8t\" (UID: \"0096d1bd-5a8d-44fd-be18-d3b269ea8107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a237e48-44z8t"
Nov 02 01:47:28.535007 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.534984    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0096d1bd-5a8d-44fd-be18-d3b269ea8107-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a237e48-44z8t\" (UID: \"0096d1bd-5a8d-44fd-be18-d3b269ea8107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a237e48-44z8t"
Nov 02 01:47:28.585472 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.585424    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bd4274e-nnf2h"]
Nov 02 01:47:28.594023 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.593974    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w96g5\" (UniqueName: \"kubernetes.io/projected/0096d1bd-5a8d-44fd-be18-d3b269ea8107-kube-api-access-w96g5\") pod \"maas-api-revoke-sub-1a237e48-44z8t\" (UID: \"0096d1bd-5a8d-44fd-be18-d3b269ea8107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a237e48-44z8t"
Nov 02 01:47:28.628588 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.628557    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 01:47:28.628751 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.628624    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52" podUID="82ab4f70-ac0f-42f7-8571-93b7ef884c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 01:47:28.633812 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.633788    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2gtk\" (UniqueName: \"kubernetes.io/projected/a30fb1a7-0467-4adc-928f-6a4a2eadb714-kube-api-access-k2gtk\") pod \"maas-api-revoke-sub-8bd4274e-nnf2h\" (UID: \"a30fb1a7-0467-4adc-928f-6a4a2eadb714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd4274e-nnf2h"
Nov 02 01:47:28.634002 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.633841    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a30fb1a7-0467-4adc-928f-6a4a2eadb714-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bd4274e-nnf2h\" (UID: \"a30fb1a7-0467-4adc-928f-6a4a2eadb714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd4274e-nnf2h"
Nov 02 01:47:28.634002 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.633912    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a30fb1a7-0467-4adc-928f-6a4a2eadb714-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bd4274e-nnf2h\" (UID: \"a30fb1a7-0467-4adc-928f-6a4a2eadb714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd4274e-nnf2h"
Nov 02 01:47:28.634685 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.634666    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a30fb1a7-0467-4adc-928f-6a4a2eadb714-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bd4274e-nnf2h\" (UID: \"a30fb1a7-0467-4adc-928f-6a4a2eadb714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd4274e-nnf2h"
Nov 02 01:47:28.636409 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.636367    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a30fb1a7-0467-4adc-928f-6a4a2eadb714-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bd4274e-nnf2h\" (UID: \"a30fb1a7-0467-4adc-928f-6a4a2eadb714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd4274e-nnf2h"
Nov 02 01:47:28.695104 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.695074    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2gtk\" (UniqueName: \"kubernetes.io/projected/a30fb1a7-0467-4adc-928f-6a4a2eadb714-kube-api-access-k2gtk\") pod \"maas-api-revoke-sub-8bd4274e-nnf2h\" (UID: \"a30fb1a7-0467-4adc-928f-6a4a2eadb714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd4274e-nnf2h"
Nov 02 01:47:28.704711 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.704688    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a237e48-44z8t"
Nov 02 01:47:28.814899 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:28.814515    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd4274e-nnf2h"
Nov 02 01:47:29.115919 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:47:29.115887    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0096d1bd_5a8d_44fd_be18_d3b269ea8107.slice/crio-9703dc36c6e45e5f22e7fce05faf6d7d15937501299d58cef503f42a55df6738 WatchSource:0}: Error finding container 9703dc36c6e45e5f22e7fce05faf6d7d15937501299d58cef503f42a55df6738: Status 404 returned error can't find the container with id 9703dc36c6e45e5f22e7fce05faf6d7d15937501299d58cef503f42a55df6738
Nov 02 01:47:29.116111 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:29.116090    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a237e48-44z8t"]
Nov 02 01:47:29.234925 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:29.234901    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bd4274e-nnf2h"]
Nov 02 01:47:29.629013 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:29.628982    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 01:47:29.629398 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:29.629027    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52" podUID="82ab4f70-ac0f-42f7-8571-93b7ef884c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 01:47:30.042196 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:30.042151    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd4274e-nnf2h" event={"ID":"a30fb1a7-0467-4adc-928f-6a4a2eadb714","Type":"ContainerStarted","Data":"d513affc5234441b3b8b09e0498c132904b32e417e5621113b949cb682f310ea"}
Nov 02 01:47:30.042449 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:30.042260    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd4274e-nnf2h" event={"ID":"a30fb1a7-0467-4adc-928f-6a4a2eadb714","Type":"ContainerStarted","Data":"c6b64f7bd9fff2be8bd0e240afc9aa6b655c5113e2627546647ea506d881f819"}
Nov 02 01:47:30.044186 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:30.044156    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a237e48-44z8t" event={"ID":"0096d1bd-5a8d-44fd-be18-d3b269ea8107","Type":"ContainerStarted","Data":"52aa9def12c3debacf4c280676b12efd523e2e4a212900ccc0cd269b217aa1f9"}
Nov 02 01:47:30.044360 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:30.044237    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a237e48-44z8t" event={"ID":"0096d1bd-5a8d-44fd-be18-d3b269ea8107","Type":"ContainerStarted","Data":"9703dc36c6e45e5f22e7fce05faf6d7d15937501299d58cef503f42a55df6738"}
Nov 02 01:47:30.187808 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:30.187733    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd4274e-nnf2h" podStartSLOduration=2.187712973 podStartE2EDuration="2.187712973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:47:30.157969787 +0000 UTC m=+1315.061651908" watchObservedRunningTime="2026-11-02 01:47:30.187712973 +0000 UTC m=+1315.091395088"
Nov 02 01:47:30.288223 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:30.288066    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a237e48-44z8t" podStartSLOduration=2.288044938 podStartE2EDuration="2.288044938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:47:30.28561358 +0000 UTC m=+1315.189295695" watchObservedRunningTime="2026-11-02 01:47:30.288044938 +0000 UTC m=+1315.191727053"
Nov 02 01:47:30.628982 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:30.628943    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 01:47:30.629146 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:30.628999    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52" podUID="82ab4f70-ac0f-42f7-8571-93b7ef884c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 01:47:31.053294 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:31.053251    5639 generic.go:356] "Generic (PLEG): container finished" podID="0096d1bd-5a8d-44fd-be18-d3b269ea8107" containerID="52aa9def12c3debacf4c280676b12efd523e2e4a212900ccc0cd269b217aa1f9" exitCode=0
Nov 02 01:47:31.053479 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:31.053333    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a237e48-44z8t" event={"ID":"0096d1bd-5a8d-44fd-be18-d3b269ea8107","Type":"ContainerDied","Data":"52aa9def12c3debacf4c280676b12efd523e2e4a212900ccc0cd269b217aa1f9"}
Nov 02 01:47:31.055036 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:31.055013    5639 generic.go:356] "Generic (PLEG): container finished" podID="a30fb1a7-0467-4adc-928f-6a4a2eadb714" containerID="d513affc5234441b3b8b09e0498c132904b32e417e5621113b949cb682f310ea" exitCode=0
Nov 02 01:47:31.055111 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:31.055092    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd4274e-nnf2h" event={"ID":"a30fb1a7-0467-4adc-928f-6a4a2eadb714","Type":"ContainerDied","Data":"d513affc5234441b3b8b09e0498c132904b32e417e5621113b949cb682f310ea"}
Nov 02 01:47:31.628654 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:31.628623    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 01:47:31.628848 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:31.628674    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52" podUID="82ab4f70-ac0f-42f7-8571-93b7ef884c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 01:47:32.278981 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:32.278950    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a237e48-44z8t"
Nov 02 01:47:32.281222 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:32.281196    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd4274e-nnf2h"
Nov 02 01:47:32.412387 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:32.412352    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a30fb1a7-0467-4adc-928f-6a4a2eadb714-kube-api-access-k2gtk\" (UniqueName: \"kubernetes.io/projected/a30fb1a7-0467-4adc-928f-6a4a2eadb714-kube-api-access-k2gtk\") pod \"a30fb1a7-0467-4adc-928f-6a4a2eadb714\" (UID: \"a30fb1a7-0467-4adc-928f-6a4a2eadb714\") "
Nov 02 01:47:32.412532 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:32.412468    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0096d1bd-5a8d-44fd-be18-d3b269ea8107-kube-api-access-w96g5\" (UniqueName: \"kubernetes.io/projected/0096d1bd-5a8d-44fd-be18-d3b269ea8107-kube-api-access-w96g5\") pod \"0096d1bd-5a8d-44fd-be18-d3b269ea8107\" (UID: \"0096d1bd-5a8d-44fd-be18-d3b269ea8107\") "
Nov 02 01:47:32.412532 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:32.412510    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0096d1bd-5a8d-44fd-be18-d3b269ea8107-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0096d1bd-5a8d-44fd-be18-d3b269ea8107-maas-api-xks-ca\") pod \"0096d1bd-5a8d-44fd-be18-d3b269ea8107\" (UID: \"0096d1bd-5a8d-44fd-be18-d3b269ea8107\") "
Nov 02 01:47:32.412624 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:32.412543    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a30fb1a7-0467-4adc-928f-6a4a2eadb714-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a30fb1a7-0467-4adc-928f-6a4a2eadb714-maas-api-openshift-ca\") pod \"a30fb1a7-0467-4adc-928f-6a4a2eadb714\" (UID: \"a30fb1a7-0467-4adc-928f-6a4a2eadb714\") "
Nov 02 01:47:32.412624 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:32.412579    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a30fb1a7-0467-4adc-928f-6a4a2eadb714-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a30fb1a7-0467-4adc-928f-6a4a2eadb714-maas-api-xks-ca\") pod \"a30fb1a7-0467-4adc-928f-6a4a2eadb714\" (UID: \"a30fb1a7-0467-4adc-928f-6a4a2eadb714\") "
Nov 02 01:47:32.412731 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:32.412716    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0096d1bd-5a8d-44fd-be18-d3b269ea8107-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0096d1bd-5a8d-44fd-be18-d3b269ea8107-maas-api-openshift-ca\") pod \"0096d1bd-5a8d-44fd-be18-d3b269ea8107\" (UID: \"0096d1bd-5a8d-44fd-be18-d3b269ea8107\") "
Nov 02 01:47:32.416040 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:32.416007    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0096d1bd-5a8d-44fd-be18-d3b269ea8107-maas-api-openshift-ca" pod "0096d1bd-5a8d-44fd-be18-d3b269ea8107" (UID: "0096d1bd-5a8d-44fd-be18-d3b269ea8107"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:47:32.418422 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:32.418389    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a30fb1a7-0467-4adc-928f-6a4a2eadb714-kube-api-access-k2gtk" pod "a30fb1a7-0467-4adc-928f-6a4a2eadb714" (UID: "a30fb1a7-0467-4adc-928f-6a4a2eadb714"). InnerVolumeSpecName "kube-api-access-k2gtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:47:32.420819 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:32.420790    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0096d1bd-5a8d-44fd-be18-d3b269ea8107-kube-api-access-w96g5" pod "0096d1bd-5a8d-44fd-be18-d3b269ea8107" (UID: "0096d1bd-5a8d-44fd-be18-d3b269ea8107"). InnerVolumeSpecName "kube-api-access-w96g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:47:32.424356 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:32.424328    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a30fb1a7-0467-4adc-928f-6a4a2eadb714-maas-api-openshift-ca" pod "a30fb1a7-0467-4adc-928f-6a4a2eadb714" (UID: "a30fb1a7-0467-4adc-928f-6a4a2eadb714"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:47:32.425151 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:32.425124    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0096d1bd-5a8d-44fd-be18-d3b269ea8107-maas-api-xks-ca" pod "0096d1bd-5a8d-44fd-be18-d3b269ea8107" (UID: "0096d1bd-5a8d-44fd-be18-d3b269ea8107"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:47:32.426704 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:32.426673    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a30fb1a7-0467-4adc-928f-6a4a2eadb714-maas-api-xks-ca" pod "a30fb1a7-0467-4adc-928f-6a4a2eadb714" (UID: "a30fb1a7-0467-4adc-928f-6a4a2eadb714"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:47:32.514553 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:32.514522    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2gtk\" (UniqueName: \"kubernetes.io/projected/a30fb1a7-0467-4adc-928f-6a4a2eadb714-kube-api-access-k2gtk\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:32.514553 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:32.514552    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w96g5\" (UniqueName: \"kubernetes.io/projected/0096d1bd-5a8d-44fd-be18-d3b269ea8107-kube-api-access-w96g5\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:32.514779 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:32.514567    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0096d1bd-5a8d-44fd-be18-d3b269ea8107-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:32.514779 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:32.514581    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a30fb1a7-0467-4adc-928f-6a4a2eadb714-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:32.514779 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:32.514595    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a30fb1a7-0467-4adc-928f-6a4a2eadb714-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:32.514779 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:32.514607    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0096d1bd-5a8d-44fd-be18-d3b269ea8107-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:32.633897 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:32.633866    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:32.634401 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:32.634377    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:32.635124 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:32.635103    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:47:33.077005 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:33.076977    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a237e48-44z8t"
Nov 02 01:47:33.077005 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:33.076981    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a237e48-44z8t" event={"ID":"0096d1bd-5a8d-44fd-be18-d3b269ea8107","Type":"ContainerDied","Data":"9703dc36c6e45e5f22e7fce05faf6d7d15937501299d58cef503f42a55df6738"}
Nov 02 01:47:33.077210 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:33.077058    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9703dc36c6e45e5f22e7fce05faf6d7d15937501299d58cef503f42a55df6738"
Nov 02 01:47:33.078567 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:33.078541    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd4274e-nnf2h" event={"ID":"a30fb1a7-0467-4adc-928f-6a4a2eadb714","Type":"ContainerDied","Data":"c6b64f7bd9fff2be8bd0e240afc9aa6b655c5113e2627546647ea506d881f819"}
Nov 02 01:47:33.078690 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:33.078573    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd4274e-nnf2h"
Nov 02 01:47:33.078690 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:33.078590    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6b64f7bd9fff2be8bd0e240afc9aa6b655c5113e2627546647ea506d881f819"
Nov 02 01:47:35.048877 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.048841    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73fc4abc-hpqtf"]
Nov 02 01:47:35.106295 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.106256    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73fc4abc-hpqtf"]
Nov 02 01:47:35.221630 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.221599    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c255371-0987-41f7-81b2-28904d6a7b47" path="/var/lib/kubelet/pods/9c255371-0987-41f7-81b2-28904d6a7b47/volumes"
Nov 02 01:47:35.430840 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.430806    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ea01c7a-svz7r"]
Nov 02 01:47:35.431453 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.431438    5639 cpu_manager.go:414] "containerMap: removing container" podUID="a30fb1a7-0467-4adc-928f-6a4a2eadb714" containerName="revoke-keys"
Nov 02 01:47:35.431525 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.431455    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a30fb1a7-0467-4adc-928f-6a4a2eadb714" containerName="revoke-keys"
Nov 02 01:47:35.431525 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.431481    5639 cpu_manager.go:414] "containerMap: removing container" podUID="0096d1bd-5a8d-44fd-be18-d3b269ea8107" containerName="revoke-keys"
Nov 02 01:47:35.431525 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.431486    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0096d1bd-5a8d-44fd-be18-d3b269ea8107" containerName="revoke-keys"
Nov 02 01:47:35.431614 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.431583    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="a30fb1a7-0467-4adc-928f-6a4a2eadb714" containerName="revoke-keys"
Nov 02 01:47:35.431614 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.431594    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="0096d1bd-5a8d-44fd-be18-d3b269ea8107" containerName="revoke-keys"
Nov 02 01:47:35.436214 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.436196    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea01c7a-svz7r"
Nov 02 01:47:35.464503 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.464471    5639 scope.go:122] "RemoveContainer" containerID="dd03b4c27a55809c8f037e3356e32d811b77f4f0d99f8806d81ba3323295981f"
Nov 02 01:47:35.492363 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.492335    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ea01c7a-svz7r"]
Nov 02 01:47:35.536867 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.536828    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:47:35.536867 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.536847    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:47:35.536867 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.536861    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:47:35.537162 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.536948    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:47:35.555864 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.555827    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgzfj\" (UniqueName: \"kubernetes.io/projected/a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0-kube-api-access-sgzfj\") pod \"maas-api-revoke-sub-5ea01c7a-svz7r\" (UID: \"a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea01c7a-svz7r"
Nov 02 01:47:35.556037 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.556015    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ea01c7a-svz7r\" (UID: \"a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea01c7a-svz7r"
Nov 02 01:47:35.556099 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.556085    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ea01c7a-svz7r\" (UID: \"a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea01c7a-svz7r"
Nov 02 01:47:35.657138 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.657096    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgzfj\" (UniqueName: \"kubernetes.io/projected/a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0-kube-api-access-sgzfj\") pod \"maas-api-revoke-sub-5ea01c7a-svz7r\" (UID: \"a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea01c7a-svz7r"
Nov 02 01:47:35.657375 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.657224    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ea01c7a-svz7r\" (UID: \"a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea01c7a-svz7r"
Nov 02 01:47:35.657375 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.657272    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ea01c7a-svz7r\" (UID: \"a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea01c7a-svz7r"
Nov 02 01:47:35.657950 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.657921    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ea01c7a-svz7r\" (UID: \"a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea01c7a-svz7r"
Nov 02 01:47:35.659837 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.659812    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ea01c7a-svz7r\" (UID: \"a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea01c7a-svz7r"
Nov 02 01:47:35.718606 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.718535    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgzfj\" (UniqueName: \"kubernetes.io/projected/a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0-kube-api-access-sgzfj\") pod \"maas-api-revoke-sub-5ea01c7a-svz7r\" (UID: \"a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea01c7a-svz7r"
Nov 02 01:47:35.750758 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:35.750731    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea01c7a-svz7r"
Nov 02 01:47:36.156301 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:36.151706    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ea01c7a-svz7r"]
Nov 02 01:47:36.209117 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:47:36.208507    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8efc2f3_fb5d_4b69_b215_7557ee5bf6b0.slice/crio-a4bd31f18c3891e3ea5ee663b40ba28684650a4207e52c767a1f17d1fb3e6063 WatchSource:0}: Error finding container a4bd31f18c3891e3ea5ee663b40ba28684650a4207e52c767a1f17d1fb3e6063: Status 404 returned error can't find the container with id a4bd31f18c3891e3ea5ee663b40ba28684650a4207e52c767a1f17d1fb3e6063
Nov 02 01:47:37.122131 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:37.122096    5639 generic.go:356] "Generic (PLEG): container finished" podID="a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0" containerID="4ba4b9968b53bdc424e033da577f8f3bf7f1050b126a8c61acb3b7c795db726c" exitCode=0
Nov 02 01:47:37.122332 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:37.122187    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea01c7a-svz7r" event={"ID":"a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0","Type":"ContainerDied","Data":"4ba4b9968b53bdc424e033da577f8f3bf7f1050b126a8c61acb3b7c795db726c"}
Nov 02 01:47:37.122332 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:37.122247    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea01c7a-svz7r" event={"ID":"a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0","Type":"ContainerStarted","Data":"a4bd31f18c3891e3ea5ee663b40ba28684650a4207e52c767a1f17d1fb3e6063"}
Nov 02 01:47:38.283552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:38.283521    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea01c7a-svz7r"
Nov 02 01:47:38.393682 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:38.393605    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0-kube-api-access-sgzfj\" (UniqueName: \"kubernetes.io/projected/a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0-kube-api-access-sgzfj\") pod \"a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0\" (UID: \"a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0\") "
Nov 02 01:47:38.393682 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:38.393638    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0-maas-api-openshift-ca\") pod \"a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0\" (UID: \"a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0\") "
Nov 02 01:47:38.393862 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:38.393757    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0-maas-api-xks-ca\") pod \"a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0\" (UID: \"a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0\") "
Nov 02 01:47:38.394065 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:38.394041    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0-maas-api-openshift-ca" pod "a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0" (UID: "a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:47:38.394529 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:38.394361    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:38.395954 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:38.395935    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0-kube-api-access-sgzfj" pod "a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0" (UID: "a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0"). InnerVolumeSpecName "kube-api-access-sgzfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:47:38.396055 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:38.396034    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0-maas-api-xks-ca" pod "a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0" (UID: "a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:47:38.495922 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:38.495886    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgzfj\" (UniqueName: \"kubernetes.io/projected/a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0-kube-api-access-sgzfj\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:38.495922 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:38.495926    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:39.106529 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.106497    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-829106b1-lssps"]
Nov 02 01:47:39.107127 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.107112    5639 cpu_manager.go:414] "containerMap: removing container" podUID="a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0" containerName="revoke-keys"
Nov 02 01:47:39.107173 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.107129    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0" containerName="revoke-keys"
Nov 02 01:47:39.107261 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.107252    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0" containerName="revoke-keys"
Nov 02 01:47:39.110946 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.110928    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-829106b1-lssps"
Nov 02 01:47:39.140776 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.140738    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea01c7a-svz7r"
Nov 02 01:47:39.140939 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.140851    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea01c7a-svz7r" event={"ID":"a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0","Type":"ContainerDied","Data":"a4bd31f18c3891e3ea5ee663b40ba28684650a4207e52c767a1f17d1fb3e6063"}
Nov 02 01:47:39.140939 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.140911    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4bd31f18c3891e3ea5ee663b40ba28684650a4207e52c767a1f17d1fb3e6063"
Nov 02 01:47:39.170138 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.170107    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"]
Nov 02 01:47:39.174484 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.174465    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-829106b1-lssps"]
Nov 02 01:47:39.174580 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.174552    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.204103 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.204066    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f6d3a51-9aeb-4548-9f8d-7158c6b13541-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-829106b1-lssps\" (UID: \"0f6d3a51-9aeb-4548-9f8d-7158c6b13541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829106b1-lssps"
Nov 02 01:47:39.204288 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.204233    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f6d3a51-9aeb-4548-9f8d-7158c6b13541-maas-api-xks-ca\") pod \"maas-api-revoke-sub-829106b1-lssps\" (UID: \"0f6d3a51-9aeb-4548-9f8d-7158c6b13541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829106b1-lssps"
Nov 02 01:47:39.204397 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.204380    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95sqf\" (UniqueName: \"kubernetes.io/projected/0f6d3a51-9aeb-4548-9f8d-7158c6b13541-kube-api-access-95sqf\") pod \"maas-api-revoke-sub-829106b1-lssps\" (UID: \"0f6d3a51-9aeb-4548-9f8d-7158c6b13541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829106b1-lssps"
Nov 02 01:47:39.231924 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.231894    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"]
Nov 02 01:47:39.305708 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.305668    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f6d3a51-9aeb-4548-9f8d-7158c6b13541-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-829106b1-lssps\" (UID: \"0f6d3a51-9aeb-4548-9f8d-7158c6b13541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829106b1-lssps"
Nov 02 01:47:39.306128 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.305757    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.306128 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.305819    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.306128 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.305982    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb887678-0155-4d81-be15-861a7e1f8ab8-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.306128 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.306022    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.306128 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.306056    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f6d3a51-9aeb-4548-9f8d-7158c6b13541-maas-api-xks-ca\") pod \"maas-api-revoke-sub-829106b1-lssps\" (UID: \"0f6d3a51-9aeb-4548-9f8d-7158c6b13541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829106b1-lssps"
Nov 02 01:47:39.306384 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.306189    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.306384 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.306252    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.306384 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.306355    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95sqf\" (UniqueName: \"kubernetes.io/projected/0f6d3a51-9aeb-4548-9f8d-7158c6b13541-kube-api-access-95sqf\") pod \"maas-api-revoke-sub-829106b1-lssps\" (UID: \"0f6d3a51-9aeb-4548-9f8d-7158c6b13541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829106b1-lssps"
Nov 02 01:47:39.306488 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.306458    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.306545 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.306483    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxnqk\" (UniqueName: \"kubernetes.io/projected/fb887678-0155-4d81-be15-861a7e1f8ab8-kube-api-access-dxnqk\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.306606 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.306543    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.306606 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.306549    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f6d3a51-9aeb-4548-9f8d-7158c6b13541-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-829106b1-lssps\" (UID: \"0f6d3a51-9aeb-4548-9f8d-7158c6b13541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829106b1-lssps"
Nov 02 01:47:39.309087 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.309063    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f6d3a51-9aeb-4548-9f8d-7158c6b13541-maas-api-xks-ca\") pod \"maas-api-revoke-sub-829106b1-lssps\" (UID: \"0f6d3a51-9aeb-4548-9f8d-7158c6b13541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829106b1-lssps"
Nov 02 01:47:39.367732 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.367662    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95sqf\" (UniqueName: \"kubernetes.io/projected/0f6d3a51-9aeb-4548-9f8d-7158c6b13541-kube-api-access-95sqf\") pod \"maas-api-revoke-sub-829106b1-lssps\" (UID: \"0f6d3a51-9aeb-4548-9f8d-7158c6b13541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829106b1-lssps"
Nov 02 01:47:39.407510 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.407471    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.407702 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.407532    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.407702 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.407567    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb887678-0155-4d81-be15-861a7e1f8ab8-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.407702 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.407591    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.407702 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.407636    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.407948 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.407713    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.407948 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.407847    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.407948 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.407875    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxnqk\" (UniqueName: \"kubernetes.io/projected/fb887678-0155-4d81-be15-861a7e1f8ab8-kube-api-access-dxnqk\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.407948 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.407923    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.408368 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.408228    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.408368 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.408325    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.408559 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.408389    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.408614 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.408586    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb887678-0155-4d81-be15-861a7e1f8ab8-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.408890 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.408865    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.410635 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.410596    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.410972 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.410953    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.426439 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.426411    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-829106b1-lssps"
Nov 02 01:47:39.468814 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.468784    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.469140 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.469116    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxnqk\" (UniqueName: \"kubernetes.io/projected/fb887678-0155-4d81-be15-861a7e1f8ab8-kube-api-access-dxnqk\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.489110 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.489079    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:39.720356 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.720257    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"]
Nov 02 01:47:39.725684 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.725664    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.786242 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.786208    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"]
Nov 02 01:47:39.814167 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.814121    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2hpp\" (UniqueName: \"kubernetes.io/projected/cbdac3f5-20af-4282-b567-66a4d880c3b4-kube-api-access-s2hpp\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.814167 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.814158    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.814393 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.814180    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.814445 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.814424    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbdac3f5-20af-4282-b567-66a4d880c3b4-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.814575 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.814546    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.814691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.814603    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.814691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.814641    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.814691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.814667    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.814828 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.814690    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.816242 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.816220    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-829106b1-lssps"]
Nov 02 01:47:39.885485 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:47:39.885446    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb887678_0155_4d81_be15_861a7e1f8ab8.slice/crio-ef0be024684bbd591431df4b605b914e5a31f5b088bfc72cebb918f93bb57518 WatchSource:0}: Error finding container ef0be024684bbd591431df4b605b914e5a31f5b088bfc72cebb918f93bb57518: Status 404 returned error can't find the container with id ef0be024684bbd591431df4b605b914e5a31f5b088bfc72cebb918f93bb57518
Nov 02 01:47:39.886789 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.886763    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"]
Nov 02 01:47:39.888093 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.888060    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:47:39.888201 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.888134    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:47:39.888201 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.888181    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:47:39.915881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.915855    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2hpp\" (UniqueName: \"kubernetes.io/projected/cbdac3f5-20af-4282-b567-66a4d880c3b4-kube-api-access-s2hpp\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.916022 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.915883    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.916022 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.915907    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.916022 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.915960    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbdac3f5-20af-4282-b567-66a4d880c3b4-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.916177 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.916041    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.916177 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.916089    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.916177 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.916125    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.916177 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.916155    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.916407 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.916184    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.916487 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.916459    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.916853 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.916814    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbdac3f5-20af-4282-b567-66a4d880c3b4-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.916917 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.916860    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.917123 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.917047    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.917368 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.917347    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.918933 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.918908    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.918933 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.918932    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.976839 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.976764    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:39.976839 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:39.976800    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2hpp\" (UniqueName: \"kubernetes.io/projected/cbdac3f5-20af-4282-b567-66a4d880c3b4-kube-api-access-s2hpp\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:40.040550 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:40.040518    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:40.160622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:40.159357    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb" event={"ID":"fb887678-0155-4d81-be15-861a7e1f8ab8","Type":"ContainerStarted","Data":"53cbe30adf15019a6faf0f6ae85b07888acec8d05bbaa9964ffb1112870ddb2e"}
Nov 02 01:47:40.160622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:40.159429    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb" event={"ID":"fb887678-0155-4d81-be15-861a7e1f8ab8","Type":"ContainerStarted","Data":"ef0be024684bbd591431df4b605b914e5a31f5b088bfc72cebb918f93bb57518"}
Nov 02 01:47:40.162937 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:40.162907    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-829106b1-lssps" event={"ID":"0f6d3a51-9aeb-4548-9f8d-7158c6b13541","Type":"ContainerStarted","Data":"f8f7af1a4da6acd0bcb4c9958da8386e847c3c8608314713352a55c99a5ce1db"}
Nov 02 01:47:40.286366 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:40.286315    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb" podStartSLOduration=1.286265265 podStartE2EDuration="1.286265265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:47:40.283695657 +0000 UTC m=+1325.187377773" watchObservedRunningTime="2026-11-02 01:47:40.286265265 +0000 UTC m=+1325.189947380"
Nov 02 01:47:40.436602 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:40.436573    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"]
Nov 02 01:47:40.437555 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:47:40.437532    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbdac3f5_20af_4282_b567_66a4d880c3b4.slice/crio-b9205aafeb821db85a3abcd35a180b7046c2e6d5f51ef35327bba43bda488126 WatchSource:0}: Error finding container b9205aafeb821db85a3abcd35a180b7046c2e6d5f51ef35327bba43bda488126: Status 404 returned error can't find the container with id b9205aafeb821db85a3abcd35a180b7046c2e6d5f51ef35327bba43bda488126
Nov 02 01:47:40.440003 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:40.439966    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:47:40.440107 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:40.440050    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:47:40.440107 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:40.440095    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:47:40.489698 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:40.489665    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:40.491638 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:40.491614    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 01:47:40.491764 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:40.491665    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb" podUID="fb887678-0155-4d81-be15-861a7e1f8ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 01:47:41.132037 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.132002    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"]
Nov 02 01:47:41.137263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.137238    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.147930 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.147906    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-workload-certs\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.148047 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.147946    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-credential-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.148047 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.147976    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-envoy\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.148137 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.148041    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f975458f-bee6-46a6-a92c-189a9509d0e6-istiod-ca-cert\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.148137 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.148078    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28vng\" (UniqueName: \"kubernetes.io/projected/f975458f-bee6-46a6-a92c-189a9509d0e6-kube-api-access-28vng\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.148137 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.148095    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-data\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.148245 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.148226    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-workload-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.148352 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.148339    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-token\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.148452 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.148435    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-podinfo\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.210039 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.209946    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw" event={"ID":"cbdac3f5-20af-4282-b567-66a4d880c3b4","Type":"ContainerStarted","Data":"c8367e6fda706d5a07d98002b1fbd959bffe814a3f89b2ea5701ecc8669eff17"}
Nov 02 01:47:41.210403 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.210383    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"]
Nov 02 01:47:41.210570 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.210552    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw" event={"ID":"cbdac3f5-20af-4282-b567-66a4d880c3b4","Type":"ContainerStarted","Data":"b9205aafeb821db85a3abcd35a180b7046c2e6d5f51ef35327bba43bda488126"}
Nov 02 01:47:41.212531 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.212499    5639 generic.go:356] "Generic (PLEG): container finished" podID="0f6d3a51-9aeb-4548-9f8d-7158c6b13541" containerID="d7835ae8e56c148a0b1e7be0ed2e4238bd9d16bc118d27a618531bdfd1946bcd" exitCode=0
Nov 02 01:47:41.224639 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.224588    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-829106b1-lssps" event={"ID":"0f6d3a51-9aeb-4548-9f8d-7158c6b13541","Type":"ContainerDied","Data":"d7835ae8e56c148a0b1e7be0ed2e4238bd9d16bc118d27a618531bdfd1946bcd"}
Nov 02 01:47:41.254845 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.253809    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-workload-certs\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.254845 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.253859    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-credential-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.254845 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.253901    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-envoy\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.254845 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.253932    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f975458f-bee6-46a6-a92c-189a9509d0e6-istiod-ca-cert\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.254845 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.254005    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28vng\" (UniqueName: \"kubernetes.io/projected/f975458f-bee6-46a6-a92c-189a9509d0e6-kube-api-access-28vng\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.254845 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.254035    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-data\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.254845 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.254103    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-workload-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.254845 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.254215    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-token\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.254845 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.254373    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-podinfo\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.255560 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.255197    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-data\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.255630 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.255567    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-workload-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.256168 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.256015    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-workload-certs\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.256863 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.256499    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-credential-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.258555 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.258130    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f975458f-bee6-46a6-a92c-189a9509d0e6-istiod-ca-cert\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.265380 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.263530    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-envoy\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.274310 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.270050    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-podinfo\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.323590 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.323548    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-token\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.325306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.325133    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28vng\" (UniqueName: \"kubernetes.io/projected/f975458f-bee6-46a6-a92c-189a9509d0e6-kube-api-access-28vng\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.401317 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.400372    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw" podStartSLOduration=2.400347928 podStartE2EDuration="2.400347928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:47:41.386798284 +0000 UTC m=+1326.290480399" watchObservedRunningTime="2026-11-02 01:47:41.400347928 +0000 UTC m=+1326.304030061"
Nov 02 01:47:41.450912 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.450885    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:41.497373 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.493718    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 01:47:41.497373 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.493797    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb" podUID="fb887678-0155-4d81-be15-861a7e1f8ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 01:47:41.964331 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:41.959043    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"]
Nov 02 01:47:42.002412 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:42.001118    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:47:42.002412 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:42.001215    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:47:42.002412 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:42.001258    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:47:42.041354 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:42.041328    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:42.042652 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:42.042627    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 01:47:42.042774 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:42.042673    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw" podUID="cbdac3f5-20af-4282-b567-66a4d880c3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 01:47:42.050877 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:42.050841    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7411f8a7-j26c2"]
Nov 02 01:47:42.055040 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:42.055016    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7411f8a7-j26c2"]
Nov 02 01:47:42.233142 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:42.233045    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs" event={"ID":"f975458f-bee6-46a6-a92c-189a9509d0e6","Type":"ContainerStarted","Data":"e984a9f7100c676ef95d95d14ad56126bb00e33caf4aa4ea6c3d3bffb3a6092c"}
Nov 02 01:47:42.233142 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:42.233114    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs" event={"ID":"f975458f-bee6-46a6-a92c-189a9509d0e6","Type":"ContainerStarted","Data":"4061b44f589099da90ab76727fc52879a73710f14d391bbf0c121561bccbc0f5"}
Nov 02 01:47:42.360314 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:42.359387    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs" podStartSLOduration=1.359365584 podStartE2EDuration="1.359365584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:47:42.35401123 +0000 UTC m=+1327.257693345" watchObservedRunningTime="2026-11-02 01:47:42.359365584 +0000 UTC m=+1327.263047698"
Nov 02 01:47:42.457051 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:42.456613    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:42.536067 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:42.535769    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 01:47:42.536067 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:42.535829    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb" podUID="fb887678-0155-4d81-be15-861a7e1f8ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 01:47:42.630184 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:42.630158    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-829106b1-lssps"
Nov 02 01:47:42.736465 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:42.736431    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f6d3a51-9aeb-4548-9f8d-7158c6b13541-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f6d3a51-9aeb-4548-9f8d-7158c6b13541-maas-api-xks-ca\") pod \"0f6d3a51-9aeb-4548-9f8d-7158c6b13541\" (UID: \"0f6d3a51-9aeb-4548-9f8d-7158c6b13541\") "
Nov 02 01:47:42.736662 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:42.736644    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f6d3a51-9aeb-4548-9f8d-7158c6b13541-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f6d3a51-9aeb-4548-9f8d-7158c6b13541-maas-api-openshift-ca\") pod \"0f6d3a51-9aeb-4548-9f8d-7158c6b13541\" (UID: \"0f6d3a51-9aeb-4548-9f8d-7158c6b13541\") "
Nov 02 01:47:42.736711 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:42.736679    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f6d3a51-9aeb-4548-9f8d-7158c6b13541-kube-api-access-95sqf\" (UniqueName: \"kubernetes.io/projected/0f6d3a51-9aeb-4548-9f8d-7158c6b13541-kube-api-access-95sqf\") pod \"0f6d3a51-9aeb-4548-9f8d-7158c6b13541\" (UID: \"0f6d3a51-9aeb-4548-9f8d-7158c6b13541\") "
Nov 02 01:47:42.736962 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:42.736935    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f6d3a51-9aeb-4548-9f8d-7158c6b13541-maas-api-openshift-ca" pod "0f6d3a51-9aeb-4548-9f8d-7158c6b13541" (UID: "0f6d3a51-9aeb-4548-9f8d-7158c6b13541"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:47:42.737293 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:42.737261    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f6d3a51-9aeb-4548-9f8d-7158c6b13541-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:42.738857 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:42.738835    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f6d3a51-9aeb-4548-9f8d-7158c6b13541-kube-api-access-95sqf" pod "0f6d3a51-9aeb-4548-9f8d-7158c6b13541" (UID: "0f6d3a51-9aeb-4548-9f8d-7158c6b13541"). InnerVolumeSpecName "kube-api-access-95sqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:47:42.739008 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:42.738992    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6d3a51-9aeb-4548-9f8d-7158c6b13541-maas-api-xks-ca" pod "0f6d3a51-9aeb-4548-9f8d-7158c6b13541" (UID: "0f6d3a51-9aeb-4548-9f8d-7158c6b13541"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:47:42.838614 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:42.838586    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f6d3a51-9aeb-4548-9f8d-7158c6b13541-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:42.838614 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:42.838613    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95sqf\" (UniqueName: \"kubernetes.io/projected/0f6d3a51-9aeb-4548-9f8d-7158c6b13541-kube-api-access-95sqf\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:43.040950 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:43.040917    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 01:47:43.041159 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:43.040974    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw" podUID="cbdac3f5-20af-4282-b567-66a4d880c3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 01:47:43.237353 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:43.236543    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2e5596f-58c4-4645-98c7-5d19ce22eb23" path="/var/lib/kubelet/pods/d2e5596f-58c4-4645-98c7-5d19ce22eb23/volumes"
Nov 02 01:47:43.251932 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:43.251908    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-829106b1-lssps"
Nov 02 01:47:43.252056 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:43.251936    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-829106b1-lssps" event={"ID":"0f6d3a51-9aeb-4548-9f8d-7158c6b13541","Type":"ContainerDied","Data":"f8f7af1a4da6acd0bcb4c9958da8386e847c3c8608314713352a55c99a5ce1db"}
Nov 02 01:47:43.252056 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:43.251981    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8f7af1a4da6acd0bcb4c9958da8386e847c3c8608314713352a55c99a5ce1db"
Nov 02 01:47:43.453053 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:43.453010    5639 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 01:47:43.453202 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:43.453069    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs" podUID="f975458f-bee6-46a6-a92c-189a9509d0e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 01:47:43.499365 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:43.499235    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 01:47:43.499365 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:43.499325    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb" podUID="fb887678-0155-4d81-be15-861a7e1f8ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 01:47:44.044858 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:44.042825    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 01:47:44.044858 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:44.042879    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw" podUID="cbdac3f5-20af-4282-b567-66a4d880c3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 01:47:44.451766 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:44.451736    5639 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 01:47:44.452025 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:44.451999    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs" podUID="f975458f-bee6-46a6-a92c-189a9509d0e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 01:47:44.495305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:44.494957    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 01:47:44.495305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:44.495017    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb" podUID="fb887678-0155-4d81-be15-861a7e1f8ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 01:47:45.041639 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:45.041596    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 01:47:45.042144 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:45.041659    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw" podUID="cbdac3f5-20af-4282-b567-66a4d880c3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 01:47:45.451542 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:45.451489    5639 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 01:47:45.451748 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:45.451555    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs" podUID="f975458f-bee6-46a6-a92c-189a9509d0e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 01:47:45.496850 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:45.496811    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 01:47:45.497173 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:45.497126    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb" podUID="fb887678-0155-4d81-be15-861a7e1f8ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 01:47:46.041304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:46.041004    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 01:47:46.041304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:46.041078    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw" podUID="cbdac3f5-20af-4282-b567-66a4d880c3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 01:47:46.452133 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:46.452100    5639 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 01:47:46.452602 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:46.452165    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs" podUID="f975458f-bee6-46a6-a92c-189a9509d0e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 01:47:46.490892 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:46.490854    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 01:47:46.491160 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:46.491133    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb" podUID="fb887678-0155-4d81-be15-861a7e1f8ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 01:47:46.830585 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:46.830554    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:47:47.041179 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:47.041135    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 01:47:47.041397 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:47.041208    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw" podUID="cbdac3f5-20af-4282-b567-66a4d880c3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 01:47:47.137383 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:47.137274    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac9cf4a5-bzpjr"]
Nov 02 01:47:47.138042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:47.138019    5639 cpu_manager.go:414] "containerMap: removing container" podUID="0f6d3a51-9aeb-4548-9f8d-7158c6b13541" containerName="revoke-keys"
Nov 02 01:47:47.138140 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:47.138043    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f6d3a51-9aeb-4548-9f8d-7158c6b13541" containerName="revoke-keys"
Nov 02 01:47:47.138225 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:47.138212    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f6d3a51-9aeb-4548-9f8d-7158c6b13541" containerName="revoke-keys"
Nov 02 01:47:47.143130 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:47.143110    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9cf4a5-bzpjr"
Nov 02 01:47:47.202232 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:47.202203    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:47:47.202444 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:47.202354    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac9cf4a5-bzpjr"]
Nov 02 01:47:47.245191 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:47.245132    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:47:47.245191 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:47.245132    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:47:47.245499 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:47.245255    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:47:47.320630 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:47.320579    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twkjw\" (UniqueName: \"kubernetes.io/projected/6b32bf8e-cc36-4898-97b0-dcda06e24f87-kube-api-access-twkjw\") pod \"maas-api-revoke-sub-ac9cf4a5-bzpjr\" (UID: \"6b32bf8e-cc36-4898-97b0-dcda06e24f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9cf4a5-bzpjr"
Nov 02 01:47:47.320842 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:47.320785    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b32bf8e-cc36-4898-97b0-dcda06e24f87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac9cf4a5-bzpjr\" (UID: \"6b32bf8e-cc36-4898-97b0-dcda06e24f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9cf4a5-bzpjr"
Nov 02 01:47:47.320907 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:47.320841    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b32bf8e-cc36-4898-97b0-dcda06e24f87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac9cf4a5-bzpjr\" (UID: \"6b32bf8e-cc36-4898-97b0-dcda06e24f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9cf4a5-bzpjr"
Nov 02 01:47:47.422233 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:47.422123    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twkjw\" (UniqueName: \"kubernetes.io/projected/6b32bf8e-cc36-4898-97b0-dcda06e24f87-kube-api-access-twkjw\") pod \"maas-api-revoke-sub-ac9cf4a5-bzpjr\" (UID: \"6b32bf8e-cc36-4898-97b0-dcda06e24f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9cf4a5-bzpjr"
Nov 02 01:47:47.422233 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:47.422218    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b32bf8e-cc36-4898-97b0-dcda06e24f87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac9cf4a5-bzpjr\" (UID: \"6b32bf8e-cc36-4898-97b0-dcda06e24f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9cf4a5-bzpjr"
Nov 02 01:47:47.422541 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:47.422253    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b32bf8e-cc36-4898-97b0-dcda06e24f87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac9cf4a5-bzpjr\" (UID: \"6b32bf8e-cc36-4898-97b0-dcda06e24f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9cf4a5-bzpjr"
Nov 02 01:47:47.423148 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:47.423117    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b32bf8e-cc36-4898-97b0-dcda06e24f87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac9cf4a5-bzpjr\" (UID: \"6b32bf8e-cc36-4898-97b0-dcda06e24f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9cf4a5-bzpjr"
Nov 02 01:47:47.424811 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:47.424789    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b32bf8e-cc36-4898-97b0-dcda06e24f87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac9cf4a5-bzpjr\" (UID: \"6b32bf8e-cc36-4898-97b0-dcda06e24f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9cf4a5-bzpjr"
Nov 02 01:47:47.452059 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:47.452018    5639 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 01:47:47.452256 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:47.452099    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs" podUID="f975458f-bee6-46a6-a92c-189a9509d0e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 01:47:47.497310 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:47.496591    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twkjw\" (UniqueName: \"kubernetes.io/projected/6b32bf8e-cc36-4898-97b0-dcda06e24f87-kube-api-access-twkjw\") pod \"maas-api-revoke-sub-ac9cf4a5-bzpjr\" (UID: \"6b32bf8e-cc36-4898-97b0-dcda06e24f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9cf4a5-bzpjr"
Nov 02 01:47:47.768354 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:47.766534    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9cf4a5-bzpjr"
Nov 02 01:47:48.044358 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:48.043743    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 01:47:48.044358 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:48.043815    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw" podUID="cbdac3f5-20af-4282-b567-66a4d880c3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 01:47:48.521306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:48.517425    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:47:48.521306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:48.517511    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb" podUID="fb887678-0155-4d81-be15-861a7e1f8ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:48.631800 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:48.631745    5639 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 01:47:48.632094 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:48.632061    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs" podUID="f975458f-bee6-46a6-a92c-189a9509d0e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 01:47:48.782574 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:48.782448    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:48.817852 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:48.817797    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:48.913652 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:48.913609    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:48.968417 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:48.968370    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:49.016915 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:49.016879    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:49.042337 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:49.042268    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:49.043122 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:49.042984    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:49.296353 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:49.235522    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:49.519653 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:49.471692    5639 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 01:47:49.519653 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:49.471752    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs" podUID="f975458f-bee6-46a6-a92c-189a9509d0e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 01:47:49.519653 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:49.472604    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 02 01:47:49.519653 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:49.475393    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:49.523045 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:49.522334    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:49.535250 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:49.534430    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:47:49.535250 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:49.534496    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb" podUID="fb887678-0155-4d81-be15-861a7e1f8ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:49.676916 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:49.665939    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac9cf4a5-bzpjr"]
Nov 02 01:47:49.789497 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:49.789438    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:47:49.793116 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:49.790994    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:47:49.832403 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:49.826389    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:49.832403 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:49.826913    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:47:49.948324 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:49.916161    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": context deadline exceeded"
Nov 02 01:47:49.948324 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:49.916589    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:47:49.984357 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:49.977137    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:49.984357 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:49.977488    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded"
Nov 02 01:47:49.984357 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:49.977608    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:47:50.086345 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:50.086272    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:47:50.139307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:50.114330    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:47:50.139307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:50.115116    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:50.139307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:50.115483    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:47:50.145837 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:50.145801    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:47:50.146074 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:50.146042    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw" podUID="cbdac3f5-20af-4282-b567-66a4d880c3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:50.286395 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:50.244748    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:50.286395 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:50.245104    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:47:50.286395 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:50.269056    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9cf4a5-bzpjr" event={"ID":"6b32bf8e-cc36-4898-97b0-dcda06e24f87","Type":"ContainerStarted","Data":"34e2b94c34d96902f5d40342f6ac3ff1a2f8d460e3abe0bd5d2b2cc6acbadb00"}
Nov 02 01:47:50.428815 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:50.380396    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:50.496308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:50.495389    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:47:50.496308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:50.495847    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:47:50.806689 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:50.798481    5639 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 01:47:50.806689 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:50.798544    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs" podUID="f975458f-bee6-46a6-a92c-189a9509d0e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 01:47:50.826309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:50.822574    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:47:50.830308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:50.826942    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:50.856305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:50.847396    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:50.856305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:50.847843    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:47:50.885308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:50.867363    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:47:50.885308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:50.867447    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb" podUID="fb887678-0155-4d81-be15-861a7e1f8ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:50.980424 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:50.968419    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": context deadline exceeded"
Nov 02 01:47:51.007799 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:50.981261    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:51.105310 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:51.094339    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:51.202316 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:51.165991    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:51.433437 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:51.192515    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:47:51.460962 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:51.460913    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw" podUID="cbdac3f5-20af-4282-b567-66a4d880c3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:51.461496 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:51.260980    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:51.504305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:51.457157    5639 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 01:47:51.504305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:51.501614    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs" podUID="f975458f-bee6-46a6-a92c-189a9509d0e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 01:47:51.504305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:51.502373    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:51.768435 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:51.768270    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:47:51.768757 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:51.768726    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5" podUID="9984c750-63ba-4996-a4b1-44549030fe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:51.852460 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:51.852411    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:51.889661 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:51.889616    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:52.076408 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:51.953240    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:52.076408 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:51.969308    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": context deadline exceeded"
Nov 02 01:47:52.076408 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:51.987032    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:52.076408 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:51.988422    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:47:52.076408 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:51.988492    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb" podUID="fb887678-0155-4d81-be15-861a7e1f8ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:52.119424 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:52.104267    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:47:52.224319 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:52.178764    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9cf4a5-bzpjr" event={"ID":"6b32bf8e-cc36-4898-97b0-dcda06e24f87","Type":"ContainerStarted","Data":"ccbd60394c6c0b2e3c0ff2b50dd2884b64af485ad7384701493ff8defa26e5ed"}
Nov 02 01:47:52.389305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:52.368514    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cdceff0-29qrc"]
Nov 02 01:47:52.483224 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:52.393496    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cdceff0-29qrc"
Nov 02 01:47:52.625685 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:52.471873    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/174ead9f-a5b5-4025-bea8-3a18dbfab699-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cdceff0-29qrc\" (UID: \"174ead9f-a5b5-4025-bea8-3a18dbfab699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cdceff0-29qrc"
Nov 02 01:47:52.625685 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:52.595367    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqnwp\" (UniqueName: \"kubernetes.io/projected/174ead9f-a5b5-4025-bea8-3a18dbfab699-kube-api-access-sqnwp\") pod \"maas-api-revoke-sub-0cdceff0-29qrc\" (UID: \"174ead9f-a5b5-4025-bea8-3a18dbfab699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cdceff0-29qrc"
Nov 02 01:47:52.625685 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:52.595698    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/174ead9f-a5b5-4025-bea8-3a18dbfab699-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cdceff0-29qrc\" (UID: \"174ead9f-a5b5-4025-bea8-3a18dbfab699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cdceff0-29qrc"
Nov 02 01:47:52.625685 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:52.482842    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:52.625685 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:52.482150    5639 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 01:47:52.625685 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:52.596815    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs" podUID="f975458f-bee6-46a6-a92c-189a9509d0e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 01:47:52.711081 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:52.482720    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:52.793406 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:52.793367    5639 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:47:52.805910 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:52.805870    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq" podUID="2da5e48a-b3fd-4b3a-97d9-505aea24dc94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:53.013362 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:52.801462    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:47:53.190434 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:53.190387    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw" podUID="cbdac3f5-20af-4282-b567-66a4d880c3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:53.190965 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:52.884542    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/174ead9f-a5b5-4025-bea8-3a18dbfab699-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cdceff0-29qrc\" (UID: \"174ead9f-a5b5-4025-bea8-3a18dbfab699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cdceff0-29qrc"
Nov 02 01:47:53.199358 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:53.191366    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqnwp\" (UniqueName: \"kubernetes.io/projected/174ead9f-a5b5-4025-bea8-3a18dbfab699-kube-api-access-sqnwp\") pod \"maas-api-revoke-sub-0cdceff0-29qrc\" (UID: \"174ead9f-a5b5-4025-bea8-3a18dbfab699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cdceff0-29qrc"
Nov 02 01:47:53.199881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:53.199849    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/174ead9f-a5b5-4025-bea8-3a18dbfab699-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cdceff0-29qrc\" (UID: \"174ead9f-a5b5-4025-bea8-3a18dbfab699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cdceff0-29qrc"
Nov 02 01:47:53.200251 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:53.019037    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:53.204733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:52.946623    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 02 01:47:53.212304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:52.973799    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded"
Nov 02 01:47:53.257313 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:52.985374    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:53.291371 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:53.258966    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/174ead9f-a5b5-4025-bea8-3a18dbfab699-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cdceff0-29qrc\" (UID: \"174ead9f-a5b5-4025-bea8-3a18dbfab699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cdceff0-29qrc"
Nov 02 01:47:53.472308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:53.266552    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:53.472308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:52.961642    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:47:53.472308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:53.457889    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cdceff0-29qrc"]
Nov 02 01:47:53.596728 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:53.596511    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:53.654314 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:47:53.645887    5639 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b32bf8e_cc36_4898_97b0_dcda06e24f87.slice/crio-conmon-ccbd60394c6c0b2e3c0ff2b50dd2884b64af485ad7384701493ff8defa26e5ed.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 01:47:53.664671 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:53.654948    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:47:53.664671 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:53.659341    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 02 01:47:53.671163 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:53.666062    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:47:53.671163 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:53.666117    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb" podUID="fb887678-0155-4d81-be15-861a7e1f8ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:53.712512 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:53.680709    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/174ead9f-a5b5-4025-bea8-3a18dbfab699-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cdceff0-29qrc\" (UID: \"174ead9f-a5b5-4025-bea8-3a18dbfab699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cdceff0-29qrc"
Nov 02 01:47:53.713400 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:53.713355    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:53.786962 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:53.782606    5639 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 01:47:53.786962 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:53.782686    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs" podUID="f975458f-bee6-46a6-a92c-189a9509d0e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 01:47:53.814701 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:53.814662    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqnwp\" (UniqueName: \"kubernetes.io/projected/174ead9f-a5b5-4025-bea8-3a18dbfab699-kube-api-access-sqnwp\") pod \"maas-api-revoke-sub-0cdceff0-29qrc\" (UID: \"174ead9f-a5b5-4025-bea8-3a18dbfab699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cdceff0-29qrc"
Nov 02 01:47:53.919433 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:53.899086    5639 generic.go:356] "Generic (PLEG): container finished" podID="6b32bf8e-cc36-4898-97b0-dcda06e24f87" containerID="ccbd60394c6c0b2e3c0ff2b50dd2884b64af485ad7384701493ff8defa26e5ed" exitCode=0
Nov 02 01:47:53.979189 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:53.977837    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:53.979189 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:53.977902    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9cf4a5-bzpjr" event={"ID":"6b32bf8e-cc36-4898-97b0-dcda06e24f87","Type":"ContainerDied","Data":"ccbd60394c6c0b2e3c0ff2b50dd2884b64af485ad7384701493ff8defa26e5ed"}
Nov 02 01:47:53.979189 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:53.979157    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:54.160315 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.154486    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:47:54.190254 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.173498    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:47:54.190254 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.175023    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:47:54.266220 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.258575    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cdceff0-29qrc"
Nov 02 01:47:54.290569 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.290513    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": context deadline exceeded"
Nov 02 01:47:54.298574 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.297910    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:54.370774 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.370744    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:47:54.379083 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.379053    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:47:54.379349 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.379320    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw" podUID="cbdac3f5-20af-4282-b567-66a4d880c3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:54.453647 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.453544    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:54.521322 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.496495    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:47:54.521908 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.521875    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:47:54.550002 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.549976    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c688bca2-b264c"]
Nov 02 01:47:54.550190 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.550179    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d24b62b-qxzwb"]
Nov 02 01:47:54.550335 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.550324    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c688bca2-b264c"]
Nov 02 01:47:54.550446 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.550437    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d24b62b-qxzwb"]
Nov 02 01:47:54.550521 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.539531    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9cf4a5-bzpjr" podStartSLOduration=7.539513103 podStartE2EDuration="7.539513103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:47:52.359363021 +0000 UTC m=+1337.263045134" watchObservedRunningTime="2026-11-02 01:47:54.539513103 +0000 UTC m=+1339.443195218"
Nov 02 01:47:54.550768 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.549212    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:54.559304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.554077    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:47:54.559304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.554126    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:47:54.599734 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.599462    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:54.633350 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.631084    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:47:54.638776 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.638751    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:47:54.654175 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.652649    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:54.654175 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.653322    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:54.654175 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.653995    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:47:54.877471 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.877433    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:54.877931 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.877910    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:54.878931 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.878913    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:47:54.937571 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:54.937536    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cdceff0-29qrc"]
Nov 02 01:47:54.940797 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:47:54.940764    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174ead9f_a5b5_4025_bea8_3a18dbfab699.slice/crio-babbc8a4ea831ddf6f443f1789372f93428aebd9e545cf49980eb5e4c00dc4c2 WatchSource:0}: Error finding container babbc8a4ea831ddf6f443f1789372f93428aebd9e545cf49980eb5e4c00dc4c2: Status 404 returned error can't find the container with id babbc8a4ea831ddf6f443f1789372f93428aebd9e545cf49980eb5e4c00dc4c2
Nov 02 01:47:55.084253 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.084224    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9cf4a5-bzpjr"
Nov 02 01:47:55.179347 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.179247    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b32bf8e-cc36-4898-97b0-dcda06e24f87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b32bf8e-cc36-4898-97b0-dcda06e24f87-maas-api-xks-ca\") pod \"6b32bf8e-cc36-4898-97b0-dcda06e24f87\" (UID: \"6b32bf8e-cc36-4898-97b0-dcda06e24f87\") "
Nov 02 01:47:55.179347 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.179324    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b32bf8e-cc36-4898-97b0-dcda06e24f87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b32bf8e-cc36-4898-97b0-dcda06e24f87-maas-api-openshift-ca\") pod \"6b32bf8e-cc36-4898-97b0-dcda06e24f87\" (UID: \"6b32bf8e-cc36-4898-97b0-dcda06e24f87\") "
Nov 02 01:47:55.179840 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.179375    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b32bf8e-cc36-4898-97b0-dcda06e24f87-kube-api-access-twkjw\" (UniqueName: \"kubernetes.io/projected/6b32bf8e-cc36-4898-97b0-dcda06e24f87-kube-api-access-twkjw\") pod \"6b32bf8e-cc36-4898-97b0-dcda06e24f87\" (UID: \"6b32bf8e-cc36-4898-97b0-dcda06e24f87\") "
Nov 02 01:47:55.179840 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.179636    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b32bf8e-cc36-4898-97b0-dcda06e24f87-maas-api-openshift-ca" pod "6b32bf8e-cc36-4898-97b0-dcda06e24f87" (UID: "6b32bf8e-cc36-4898-97b0-dcda06e24f87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:47:55.180175 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.180144    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b32bf8e-cc36-4898-97b0-dcda06e24f87-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:55.182341 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.182316    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b32bf8e-cc36-4898-97b0-dcda06e24f87-maas-api-xks-ca" pod "6b32bf8e-cc36-4898-97b0-dcda06e24f87" (UID: "6b32bf8e-cc36-4898-97b0-dcda06e24f87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:47:55.182456 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.182343    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b32bf8e-cc36-4898-97b0-dcda06e24f87-kube-api-access-twkjw" pod "6b32bf8e-cc36-4898-97b0-dcda06e24f87" (UID: "6b32bf8e-cc36-4898-97b0-dcda06e24f87"). InnerVolumeSpecName "kube-api-access-twkjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:47:55.222235 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.222202    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="679a115f-0425-4323-babf-8c2697236035" path="/var/lib/kubelet/pods/679a115f-0425-4323-babf-8c2697236035/volumes"
Nov 02 01:47:55.226995 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.226397    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4" path="/var/lib/kubelet/pods/933a125f-f96c-4f9f-a7d6-7a6e9bdeaee4/volumes"
Nov 02 01:47:55.281133 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.281106    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b32bf8e-cc36-4898-97b0-dcda06e24f87-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:55.281625 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.281135    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twkjw\" (UniqueName: \"kubernetes.io/projected/6b32bf8e-cc36-4898-97b0-dcda06e24f87-kube-api-access-twkjw\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:55.502582 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.502552    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5c9e24a-c7gkz"]
Nov 02 01:47:55.503175 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.503162    5639 cpu_manager.go:414] "containerMap: removing container" podUID="6b32bf8e-cc36-4898-97b0-dcda06e24f87" containerName="revoke-keys"
Nov 02 01:47:55.503228 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.503177    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b32bf8e-cc36-4898-97b0-dcda06e24f87" containerName="revoke-keys"
Nov 02 01:47:55.503331 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.503321    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b32bf8e-cc36-4898-97b0-dcda06e24f87" containerName="revoke-keys"
Nov 02 01:47:55.506926 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.506908    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c9e24a-c7gkz"
Nov 02 01:47:55.571644 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.571614    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5c9e24a-c7gkz"]
Nov 02 01:47:55.585876 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.585838    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jsvv\" (UniqueName: \"kubernetes.io/projected/05d86e93-5bbf-40ee-b1e8-a20c811378ba-kube-api-access-7jsvv\") pod \"maas-api-revoke-sub-e5c9e24a-c7gkz\" (UID: \"05d86e93-5bbf-40ee-b1e8-a20c811378ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c9e24a-c7gkz"
Nov 02 01:47:55.586076 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.586055    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05d86e93-5bbf-40ee-b1e8-a20c811378ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5c9e24a-c7gkz\" (UID: \"05d86e93-5bbf-40ee-b1e8-a20c811378ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c9e24a-c7gkz"
Nov 02 01:47:55.586150 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.586118    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05d86e93-5bbf-40ee-b1e8-a20c811378ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5c9e24a-c7gkz\" (UID: \"05d86e93-5bbf-40ee-b1e8-a20c811378ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c9e24a-c7gkz"
Nov 02 01:47:55.687871 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.687841    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05d86e93-5bbf-40ee-b1e8-a20c811378ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5c9e24a-c7gkz\" (UID: \"05d86e93-5bbf-40ee-b1e8-a20c811378ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c9e24a-c7gkz"
Nov 02 01:47:55.688067 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.687901    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05d86e93-5bbf-40ee-b1e8-a20c811378ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5c9e24a-c7gkz\" (UID: \"05d86e93-5bbf-40ee-b1e8-a20c811378ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c9e24a-c7gkz"
Nov 02 01:47:55.688067 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.688023    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jsvv\" (UniqueName: \"kubernetes.io/projected/05d86e93-5bbf-40ee-b1e8-a20c811378ba-kube-api-access-7jsvv\") pod \"maas-api-revoke-sub-e5c9e24a-c7gkz\" (UID: \"05d86e93-5bbf-40ee-b1e8-a20c811378ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c9e24a-c7gkz"
Nov 02 01:47:55.688974 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.688948    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05d86e93-5bbf-40ee-b1e8-a20c811378ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5c9e24a-c7gkz\" (UID: \"05d86e93-5bbf-40ee-b1e8-a20c811378ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c9e24a-c7gkz"
Nov 02 01:47:55.690562 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.690538    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05d86e93-5bbf-40ee-b1e8-a20c811378ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5c9e24a-c7gkz\" (UID: \"05d86e93-5bbf-40ee-b1e8-a20c811378ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c9e24a-c7gkz"
Nov 02 01:47:55.748919 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.748852    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jsvv\" (UniqueName: \"kubernetes.io/projected/05d86e93-5bbf-40ee-b1e8-a20c811378ba-kube-api-access-7jsvv\") pod \"maas-api-revoke-sub-e5c9e24a-c7gkz\" (UID: \"05d86e93-5bbf-40ee-b1e8-a20c811378ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c9e24a-c7gkz"
Nov 02 01:47:55.820859 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.820831    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c9e24a-c7gkz"
Nov 02 01:47:55.940325 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.937999    5639 generic.go:356] "Generic (PLEG): container finished" podID="174ead9f-a5b5-4025-bea8-3a18dbfab699" containerID="da588cf5782dd1ab6937881732c33253109b92e3d5c104f0a92db225266365e9" exitCode=0
Nov 02 01:47:55.940325 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.938135    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cdceff0-29qrc" event={"ID":"174ead9f-a5b5-4025-bea8-3a18dbfab699","Type":"ContainerDied","Data":"da588cf5782dd1ab6937881732c33253109b92e3d5c104f0a92db225266365e9"}
Nov 02 01:47:55.940325 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.938189    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cdceff0-29qrc" event={"ID":"174ead9f-a5b5-4025-bea8-3a18dbfab699","Type":"ContainerStarted","Data":"babbc8a4ea831ddf6f443f1789372f93428aebd9e545cf49980eb5e4c00dc4c2"}
Nov 02 01:47:55.941965 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.941946    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9cf4a5-bzpjr"
Nov 02 01:47:55.942412 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.942385    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9cf4a5-bzpjr" event={"ID":"6b32bf8e-cc36-4898-97b0-dcda06e24f87","Type":"ContainerDied","Data":"34e2b94c34d96902f5d40342f6ac3ff1a2f8d460e3abe0bd5d2b2cc6acbadb00"}
Nov 02 01:47:55.942539 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:55.942421    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34e2b94c34d96902f5d40342f6ac3ff1a2f8d460e3abe0bd5d2b2cc6acbadb00"
Nov 02 01:47:56.327965 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:56.327940    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5c9e24a-c7gkz"]
Nov 02 01:47:56.328715 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:47:56.328687    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d86e93_5bbf_40ee_b1e8_a20c811378ba.slice/crio-6215b931fe8ec57c0946635b8d79baeecc0a6e4c5bfd7a79e763ccb69851687c WatchSource:0}: Error finding container 6215b931fe8ec57c0946635b8d79baeecc0a6e4c5bfd7a79e763ccb69851687c: Status 404 returned error can't find the container with id 6215b931fe8ec57c0946635b8d79baeecc0a6e4c5bfd7a79e763ccb69851687c
Nov 02 01:47:56.951798 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:56.951707    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c9e24a-c7gkz" event={"ID":"05d86e93-5bbf-40ee-b1e8-a20c811378ba","Type":"ContainerStarted","Data":"05b0aa7e692003c7df5be175feb0d8ab24d07b14c2c83b38bac732565786d45e"}
Nov 02 01:47:56.951798 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:56.951769    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c9e24a-c7gkz" event={"ID":"05d86e93-5bbf-40ee-b1e8-a20c811378ba","Type":"ContainerStarted","Data":"6215b931fe8ec57c0946635b8d79baeecc0a6e4c5bfd7a79e763ccb69851687c"}
Nov 02 01:47:57.072899 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.072846    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c9e24a-c7gkz" podStartSLOduration=2.072820532 podStartE2EDuration="2.072820532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:47:57.070800924 +0000 UTC m=+1341.974483036" watchObservedRunningTime="2026-11-02 01:47:57.072820532 +0000 UTC m=+1341.976502669"
Nov 02 01:47:57.114788 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.114756    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cdceff0-29qrc"
Nov 02 01:47:57.195311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.194388    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-346c5448-s542p"]
Nov 02 01:47:57.203492 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.195810    5639 cpu_manager.go:414] "containerMap: removing container" podUID="174ead9f-a5b5-4025-bea8-3a18dbfab699" containerName="revoke-keys"
Nov 02 01:47:57.203492 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.195835    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="174ead9f-a5b5-4025-bea8-3a18dbfab699" containerName="revoke-keys"
Nov 02 01:47:57.203492 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.196236    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="174ead9f-a5b5-4025-bea8-3a18dbfab699" containerName="revoke-keys"
Nov 02 01:47:57.204584 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.204561    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-346c5448-s542p"
Nov 02 01:47:57.213661 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.213632    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/174ead9f-a5b5-4025-bea8-3a18dbfab699-kube-api-access-sqnwp\" (UniqueName: \"kubernetes.io/projected/174ead9f-a5b5-4025-bea8-3a18dbfab699-kube-api-access-sqnwp\") pod \"174ead9f-a5b5-4025-bea8-3a18dbfab699\" (UID: \"174ead9f-a5b5-4025-bea8-3a18dbfab699\") "
Nov 02 01:47:57.213758 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.213709    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/174ead9f-a5b5-4025-bea8-3a18dbfab699-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/174ead9f-a5b5-4025-bea8-3a18dbfab699-maas-api-openshift-ca\") pod \"174ead9f-a5b5-4025-bea8-3a18dbfab699\" (UID: \"174ead9f-a5b5-4025-bea8-3a18dbfab699\") "
Nov 02 01:47:57.213758 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.213745    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/174ead9f-a5b5-4025-bea8-3a18dbfab699-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/174ead9f-a5b5-4025-bea8-3a18dbfab699-maas-api-xks-ca\") pod \"174ead9f-a5b5-4025-bea8-3a18dbfab699\" (UID: \"174ead9f-a5b5-4025-bea8-3a18dbfab699\") "
Nov 02 01:47:57.214359 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.214331    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/174ead9f-a5b5-4025-bea8-3a18dbfab699-maas-api-openshift-ca" pod "174ead9f-a5b5-4025-bea8-3a18dbfab699" (UID: "174ead9f-a5b5-4025-bea8-3a18dbfab699"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:47:57.214662 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.214640    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/174ead9f-a5b5-4025-bea8-3a18dbfab699-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:57.216521 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.216495    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/174ead9f-a5b5-4025-bea8-3a18dbfab699-maas-api-xks-ca" pod "174ead9f-a5b5-4025-bea8-3a18dbfab699" (UID: "174ead9f-a5b5-4025-bea8-3a18dbfab699"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:47:57.216661 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.216640    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/174ead9f-a5b5-4025-bea8-3a18dbfab699-kube-api-access-sqnwp" pod "174ead9f-a5b5-4025-bea8-3a18dbfab699" (UID: "174ead9f-a5b5-4025-bea8-3a18dbfab699"). InnerVolumeSpecName "kube-api-access-sqnwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:47:57.254420 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.254388    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-346c5448-s542p"]
Nov 02 01:47:57.315961 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.315924    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a5cb8e-fafb-42bc-a01a-fbb88873a88b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-346c5448-s542p\" (UID: \"f0a5cb8e-fafb-42bc-a01a-fbb88873a88b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346c5448-s542p"
Nov 02 01:47:57.316110 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.316050    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f7q4\" (UniqueName: \"kubernetes.io/projected/f0a5cb8e-fafb-42bc-a01a-fbb88873a88b-kube-api-access-5f7q4\") pod \"maas-api-revoke-sub-346c5448-s542p\" (UID: \"f0a5cb8e-fafb-42bc-a01a-fbb88873a88b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346c5448-s542p"
Nov 02 01:47:57.316230 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.316213    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a5cb8e-fafb-42bc-a01a-fbb88873a88b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-346c5448-s542p\" (UID: \"f0a5cb8e-fafb-42bc-a01a-fbb88873a88b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346c5448-s542p"
Nov 02 01:47:57.316394 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.316380    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqnwp\" (UniqueName: \"kubernetes.io/projected/174ead9f-a5b5-4025-bea8-3a18dbfab699-kube-api-access-sqnwp\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:57.316463 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.316400    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/174ead9f-a5b5-4025-bea8-3a18dbfab699-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:57.417996 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.417963    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a5cb8e-fafb-42bc-a01a-fbb88873a88b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-346c5448-s542p\" (UID: \"f0a5cb8e-fafb-42bc-a01a-fbb88873a88b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346c5448-s542p"
Nov 02 01:47:57.418505 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.418097    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a5cb8e-fafb-42bc-a01a-fbb88873a88b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-346c5448-s542p\" (UID: \"f0a5cb8e-fafb-42bc-a01a-fbb88873a88b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346c5448-s542p"
Nov 02 01:47:57.418505 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.418149    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f7q4\" (UniqueName: \"kubernetes.io/projected/f0a5cb8e-fafb-42bc-a01a-fbb88873a88b-kube-api-access-5f7q4\") pod \"maas-api-revoke-sub-346c5448-s542p\" (UID: \"f0a5cb8e-fafb-42bc-a01a-fbb88873a88b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346c5448-s542p"
Nov 02 01:47:57.419010 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.418946    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a5cb8e-fafb-42bc-a01a-fbb88873a88b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-346c5448-s542p\" (UID: \"f0a5cb8e-fafb-42bc-a01a-fbb88873a88b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346c5448-s542p"
Nov 02 01:47:57.420641 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.420617    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a5cb8e-fafb-42bc-a01a-fbb88873a88b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-346c5448-s542p\" (UID: \"f0a5cb8e-fafb-42bc-a01a-fbb88873a88b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346c5448-s542p"
Nov 02 01:47:57.422845 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:47:57.422817    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:47:57.478998 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.478929    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f7q4\" (UniqueName: \"kubernetes.io/projected/f0a5cb8e-fafb-42bc-a01a-fbb88873a88b-kube-api-access-5f7q4\") pod \"maas-api-revoke-sub-346c5448-s542p\" (UID: \"f0a5cb8e-fafb-42bc-a01a-fbb88873a88b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346c5448-s542p"
Nov 02 01:47:57.521448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.521418    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-346c5448-s542p"
Nov 02 01:47:57.904409 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.904379    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-346c5448-s542p"]
Nov 02 01:47:57.904795 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:47:57.904770    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0a5cb8e_fafb_42bc_a01a_fbb88873a88b.slice/crio-23ca636884dab1a98883d0a91287457781a5350e03761bd06274aadbdd289526 WatchSource:0}: Error finding container 23ca636884dab1a98883d0a91287457781a5350e03761bd06274aadbdd289526: Status 404 returned error can't find the container with id 23ca636884dab1a98883d0a91287457781a5350e03761bd06274aadbdd289526
Nov 02 01:47:57.979442 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.979393    5639 generic.go:356] "Generic (PLEG): container finished" podID="05d86e93-5bbf-40ee-b1e8-a20c811378ba" containerID="05b0aa7e692003c7df5be175feb0d8ab24d07b14c2c83b38bac732565786d45e" exitCode=0
Nov 02 01:47:57.979625 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.979486    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c9e24a-c7gkz" event={"ID":"05d86e93-5bbf-40ee-b1e8-a20c811378ba","Type":"ContainerDied","Data":"05b0aa7e692003c7df5be175feb0d8ab24d07b14c2c83b38bac732565786d45e"}
Nov 02 01:47:57.981574 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.981544    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-346c5448-s542p" event={"ID":"f0a5cb8e-fafb-42bc-a01a-fbb88873a88b","Type":"ContainerStarted","Data":"23ca636884dab1a98883d0a91287457781a5350e03761bd06274aadbdd289526"}
Nov 02 01:47:57.986047 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.986024    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cdceff0-29qrc"
Nov 02 01:47:57.986154 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.986049    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cdceff0-29qrc" event={"ID":"174ead9f-a5b5-4025-bea8-3a18dbfab699","Type":"ContainerDied","Data":"babbc8a4ea831ddf6f443f1789372f93428aebd9e545cf49980eb5e4c00dc4c2"}
Nov 02 01:47:57.986154 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:57.986104    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="babbc8a4ea831ddf6f443f1789372f93428aebd9e545cf49980eb5e4c00dc4c2"
Nov 02 01:47:59.020161 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:59.020059    5639 generic.go:356] "Generic (PLEG): container finished" podID="f0a5cb8e-fafb-42bc-a01a-fbb88873a88b" containerID="eb16ce431579c03ae2cbda2f4fa7f840074007c081324cd684ac24c103f7a9a5" exitCode=0
Nov 02 01:47:59.021088 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:59.021047    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-346c5448-s542p" event={"ID":"f0a5cb8e-fafb-42bc-a01a-fbb88873a88b","Type":"ContainerDied","Data":"eb16ce431579c03ae2cbda2f4fa7f840074007c081324cd684ac24c103f7a9a5"}
Nov 02 01:47:59.201998 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:59.201970    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c9e24a-c7gkz"
Nov 02 01:47:59.249640 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:59.249614    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05d86e93-5bbf-40ee-b1e8-a20c811378ba-kube-api-access-7jsvv\" (UniqueName: \"kubernetes.io/projected/05d86e93-5bbf-40ee-b1e8-a20c811378ba-kube-api-access-7jsvv\") pod \"05d86e93-5bbf-40ee-b1e8-a20c811378ba\" (UID: \"05d86e93-5bbf-40ee-b1e8-a20c811378ba\") "
Nov 02 01:47:59.249793 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:59.249778    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05d86e93-5bbf-40ee-b1e8-a20c811378ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05d86e93-5bbf-40ee-b1e8-a20c811378ba-maas-api-openshift-ca\") pod \"05d86e93-5bbf-40ee-b1e8-a20c811378ba\" (UID: \"05d86e93-5bbf-40ee-b1e8-a20c811378ba\") "
Nov 02 01:47:59.249849 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:59.249837    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05d86e93-5bbf-40ee-b1e8-a20c811378ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05d86e93-5bbf-40ee-b1e8-a20c811378ba-maas-api-xks-ca\") pod \"05d86e93-5bbf-40ee-b1e8-a20c811378ba\" (UID: \"05d86e93-5bbf-40ee-b1e8-a20c811378ba\") "
Nov 02 01:47:59.250153 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:59.250116    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05d86e93-5bbf-40ee-b1e8-a20c811378ba-maas-api-openshift-ca" pod "05d86e93-5bbf-40ee-b1e8-a20c811378ba" (UID: "05d86e93-5bbf-40ee-b1e8-a20c811378ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:47:59.250414 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:59.250395    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05d86e93-5bbf-40ee-b1e8-a20c811378ba-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:59.252079 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:59.252056    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05d86e93-5bbf-40ee-b1e8-a20c811378ba-maas-api-xks-ca" pod "05d86e93-5bbf-40ee-b1e8-a20c811378ba" (UID: "05d86e93-5bbf-40ee-b1e8-a20c811378ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:47:59.252221 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:59.252202    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d86e93-5bbf-40ee-b1e8-a20c811378ba-kube-api-access-7jsvv" pod "05d86e93-5bbf-40ee-b1e8-a20c811378ba" (UID: "05d86e93-5bbf-40ee-b1e8-a20c811378ba"). InnerVolumeSpecName "kube-api-access-7jsvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:47:59.358736 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:59.358694    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05d86e93-5bbf-40ee-b1e8-a20c811378ba-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:59.358736 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:47:59.358738    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jsvv\" (UniqueName: \"kubernetes.io/projected/05d86e93-5bbf-40ee-b1e8-a20c811378ba-kube-api-access-7jsvv\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:00.029162 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:00.029131    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c9e24a-c7gkz"
Nov 02 01:48:00.029553 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:00.029170    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c9e24a-c7gkz" event={"ID":"05d86e93-5bbf-40ee-b1e8-a20c811378ba","Type":"ContainerDied","Data":"6215b931fe8ec57c0946635b8d79baeecc0a6e4c5bfd7a79e763ccb69851687c"}
Nov 02 01:48:00.029553 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:00.029218    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6215b931fe8ec57c0946635b8d79baeecc0a6e4c5bfd7a79e763ccb69851687c"
Nov 02 01:48:00.189908 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:00.189820    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-346c5448-s542p"
Nov 02 01:48:00.271583 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:00.271547    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0a5cb8e-fafb-42bc-a01a-fbb88873a88b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a5cb8e-fafb-42bc-a01a-fbb88873a88b-maas-api-openshift-ca\") pod \"f0a5cb8e-fafb-42bc-a01a-fbb88873a88b\" (UID: \"f0a5cb8e-fafb-42bc-a01a-fbb88873a88b\") "
Nov 02 01:48:00.271739 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:00.271658    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0a5cb8e-fafb-42bc-a01a-fbb88873a88b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a5cb8e-fafb-42bc-a01a-fbb88873a88b-maas-api-xks-ca\") pod \"f0a5cb8e-fafb-42bc-a01a-fbb88873a88b\" (UID: \"f0a5cb8e-fafb-42bc-a01a-fbb88873a88b\") "
Nov 02 01:48:00.271739 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:00.271710    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0a5cb8e-fafb-42bc-a01a-fbb88873a88b-kube-api-access-5f7q4\" (UniqueName: \"kubernetes.io/projected/f0a5cb8e-fafb-42bc-a01a-fbb88873a88b-kube-api-access-5f7q4\") pod \"f0a5cb8e-fafb-42bc-a01a-fbb88873a88b\" (UID: \"f0a5cb8e-fafb-42bc-a01a-fbb88873a88b\") "
Nov 02 01:48:00.271976 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:00.271950    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a5cb8e-fafb-42bc-a01a-fbb88873a88b-maas-api-openshift-ca" pod "f0a5cb8e-fafb-42bc-a01a-fbb88873a88b" (UID: "f0a5cb8e-fafb-42bc-a01a-fbb88873a88b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:00.272161 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:00.272146    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a5cb8e-fafb-42bc-a01a-fbb88873a88b-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:00.273856 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:00.273828    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0a5cb8e-fafb-42bc-a01a-fbb88873a88b-maas-api-xks-ca" pod "f0a5cb8e-fafb-42bc-a01a-fbb88873a88b" (UID: "f0a5cb8e-fafb-42bc-a01a-fbb88873a88b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:00.273980 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:00.273880    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a5cb8e-fafb-42bc-a01a-fbb88873a88b-kube-api-access-5f7q4" pod "f0a5cb8e-fafb-42bc-a01a-fbb88873a88b" (UID: "f0a5cb8e-fafb-42bc-a01a-fbb88873a88b"). InnerVolumeSpecName "kube-api-access-5f7q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:00.373739 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:00.373691    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a5cb8e-fafb-42bc-a01a-fbb88873a88b-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:00.373739 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:00.373728    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f7q4\" (UniqueName: \"kubernetes.io/projected/f0a5cb8e-fafb-42bc-a01a-fbb88873a88b-kube-api-access-5f7q4\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:01.033505 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:01.033459    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6ed0482-2mkqb"]
Nov 02 01:48:01.034266 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:01.034244    5639 cpu_manager.go:414] "containerMap: removing container" podUID="f0a5cb8e-fafb-42bc-a01a-fbb88873a88b" containerName="revoke-keys"
Nov 02 01:48:01.034450 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:01.034268    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0a5cb8e-fafb-42bc-a01a-fbb88873a88b" containerName="revoke-keys"
Nov 02 01:48:01.034450 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:01.034381    5639 cpu_manager.go:414] "containerMap: removing container" podUID="05d86e93-5bbf-40ee-b1e8-a20c811378ba" containerName="revoke-keys"
Nov 02 01:48:01.034450 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:01.034392    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05d86e93-5bbf-40ee-b1e8-a20c811378ba" containerName="revoke-keys"
Nov 02 01:48:01.034605 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:01.034572    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="05d86e93-5bbf-40ee-b1e8-a20c811378ba" containerName="revoke-keys"
Nov 02 01:48:01.034605 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:01.034592    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0a5cb8e-fafb-42bc-a01a-fbb88873a88b" containerName="revoke-keys"
Nov 02 01:48:01.038417 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:01.038398    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ed0482-2mkqb"
Nov 02 01:48:01.038546 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:01.038502    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-346c5448-s542p" event={"ID":"f0a5cb8e-fafb-42bc-a01a-fbb88873a88b","Type":"ContainerDied","Data":"23ca636884dab1a98883d0a91287457781a5350e03761bd06274aadbdd289526"}
Nov 02 01:48:01.038546 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:01.038528    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-346c5448-s542p"
Nov 02 01:48:01.038546 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:01.038545    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23ca636884dab1a98883d0a91287457781a5350e03761bd06274aadbdd289526"
Nov 02 01:48:01.063237 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:01.063211    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-667f49cf-k6mxc"]
Nov 02 01:48:01.082913 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:01.082880    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-667f49cf-k6mxc"]
Nov 02 01:48:01.094967 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:01.094940    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6ed0482-2mkqb"]
Nov 02 01:48:01.184410 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:01.184381    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgtn8\" (UniqueName: \"kubernetes.io/projected/9726c116-9266-4a62-bfd7-3d58ddb6cf98-kube-api-access-vgtn8\") pod \"maas-api-revoke-sub-b6ed0482-2mkqb\" (UID: \"9726c116-9266-4a62-bfd7-3d58ddb6cf98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ed0482-2mkqb"
Nov 02 01:48:01.184410 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:01.184414    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9726c116-9266-4a62-bfd7-3d58ddb6cf98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6ed0482-2mkqb\" (UID: \"9726c116-9266-4a62-bfd7-3d58ddb6cf98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ed0482-2mkqb"
Nov 02 01:48:01.184646 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:01.184439    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9726c116-9266-4a62-bfd7-3d58ddb6cf98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6ed0482-2mkqb\" (UID: \"9726c116-9266-4a62-bfd7-3d58ddb6cf98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ed0482-2mkqb"
Nov 02 01:48:01.221137 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:01.221102    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c909f3b-9a71-4f93-838c-1e86d9c41897" path="/var/lib/kubelet/pods/9c909f3b-9a71-4f93-838c-1e86d9c41897/volumes"
Nov 02 01:48:01.286026 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:01.285941    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9726c116-9266-4a62-bfd7-3d58ddb6cf98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6ed0482-2mkqb\" (UID: \"9726c116-9266-4a62-bfd7-3d58ddb6cf98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ed0482-2mkqb"
Nov 02 01:48:01.286026 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:01.285988    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9726c116-9266-4a62-bfd7-3d58ddb6cf98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6ed0482-2mkqb\" (UID: \"9726c116-9266-4a62-bfd7-3d58ddb6cf98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ed0482-2mkqb"
Nov 02 01:48:01.286363 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:01.286325    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgtn8\" (UniqueName: \"kubernetes.io/projected/9726c116-9266-4a62-bfd7-3d58ddb6cf98-kube-api-access-vgtn8\") pod \"maas-api-revoke-sub-b6ed0482-2mkqb\" (UID: \"9726c116-9266-4a62-bfd7-3d58ddb6cf98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ed0482-2mkqb"
Nov 02 01:48:01.286793 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:01.286768    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9726c116-9266-4a62-bfd7-3d58ddb6cf98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6ed0482-2mkqb\" (UID: \"9726c116-9266-4a62-bfd7-3d58ddb6cf98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ed0482-2mkqb"
Nov 02 01:48:01.288591 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:01.288574    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9726c116-9266-4a62-bfd7-3d58ddb6cf98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6ed0482-2mkqb\" (UID: \"9726c116-9266-4a62-bfd7-3d58ddb6cf98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ed0482-2mkqb"
Nov 02 01:48:01.347340 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:01.347310    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgtn8\" (UniqueName: \"kubernetes.io/projected/9726c116-9266-4a62-bfd7-3d58ddb6cf98-kube-api-access-vgtn8\") pod \"maas-api-revoke-sub-b6ed0482-2mkqb\" (UID: \"9726c116-9266-4a62-bfd7-3d58ddb6cf98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ed0482-2mkqb"
Nov 02 01:48:01.354206 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:01.354178    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ed0482-2mkqb"
Nov 02 01:48:01.732927 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:01.732898    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6ed0482-2mkqb"]
Nov 02 01:48:01.733095 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:48:01.733073    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9726c116_9266_4a62_bfd7_3d58ddb6cf98.slice/crio-1f37004020297bb25776c46261c94ed312801ca17843a4a167638637501858f9 WatchSource:0}: Error finding container 1f37004020297bb25776c46261c94ed312801ca17843a4a167638637501858f9: Status 404 returned error can't find the container with id 1f37004020297bb25776c46261c94ed312801ca17843a4a167638637501858f9
Nov 02 01:48:02.048457 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:02.048378    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ed0482-2mkqb" event={"ID":"9726c116-9266-4a62-bfd7-3d58ddb6cf98","Type":"ContainerStarted","Data":"1f37004020297bb25776c46261c94ed312801ca17843a4a167638637501858f9"}
Nov 02 01:48:02.056914 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:02.056874    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9347a0a-7g2r8"]
Nov 02 01:48:02.062721 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:02.062686    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9347a0a-7g2r8"]
Nov 02 01:48:03.049651 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:03.049571    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fc9016b-lx48f"]
Nov 02 01:48:03.062758 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:03.062726    5639 generic.go:356] "Generic (PLEG): container finished" podID="9726c116-9266-4a62-bfd7-3d58ddb6cf98" containerID="a0cf4436ee4cca56be1c87491c13df298da4d36a20bcd866236799a7d7ba5d72" exitCode=0
Nov 02 01:48:03.062933 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:03.062815    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ed0482-2mkqb" event={"ID":"9726c116-9266-4a62-bfd7-3d58ddb6cf98","Type":"ContainerDied","Data":"a0cf4436ee4cca56be1c87491c13df298da4d36a20bcd866236799a7d7ba5d72"}
Nov 02 01:48:03.101776 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:03.101739    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fc9016b-lx48f"]
Nov 02 01:48:03.221735 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:03.221701    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67cf0050-d217-4871-aa43-afab78812551" path="/var/lib/kubelet/pods/67cf0050-d217-4871-aa43-afab78812551/volumes"
Nov 02 01:48:03.222260 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:03.222237    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a385d2ad-22c8-4f1e-b98b-9ead1eb4019d" path="/var/lib/kubelet/pods/a385d2ad-22c8-4f1e-b98b-9ead1eb4019d/volumes"
Nov 02 01:48:04.157348 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:04.153849    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f1b79b0-5q5hw"]
Nov 02 01:48:04.159748 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:04.159720    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1b79b0-5q5hw"
Nov 02 01:48:04.217021 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:04.216991    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f1b79b0-5q5hw"]
Nov 02 01:48:04.229832 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:04.229759    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f96ea7a-c7f4-429c-9c49-39a99091044f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f1b79b0-5q5hw\" (UID: \"2f96ea7a-c7f4-429c-9c49-39a99091044f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1b79b0-5q5hw"
Nov 02 01:48:04.229998 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:04.229975    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt8bh\" (UniqueName: \"kubernetes.io/projected/2f96ea7a-c7f4-429c-9c49-39a99091044f-kube-api-access-mt8bh\") pod \"maas-api-revoke-sub-6f1b79b0-5q5hw\" (UID: \"2f96ea7a-c7f4-429c-9c49-39a99091044f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1b79b0-5q5hw"
Nov 02 01:48:04.230322 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:04.230138    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f96ea7a-c7f4-429c-9c49-39a99091044f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f1b79b0-5q5hw\" (UID: \"2f96ea7a-c7f4-429c-9c49-39a99091044f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1b79b0-5q5hw"
Nov 02 01:48:04.251203 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:04.251178    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ed0482-2mkqb"
Nov 02 01:48:04.331706 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:04.331677    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9726c116-9266-4a62-bfd7-3d58ddb6cf98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9726c116-9266-4a62-bfd7-3d58ddb6cf98-maas-api-xks-ca\") pod \"9726c116-9266-4a62-bfd7-3d58ddb6cf98\" (UID: \"9726c116-9266-4a62-bfd7-3d58ddb6cf98\") "
Nov 02 01:48:04.331889 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:04.331804    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9726c116-9266-4a62-bfd7-3d58ddb6cf98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9726c116-9266-4a62-bfd7-3d58ddb6cf98-maas-api-openshift-ca\") pod \"9726c116-9266-4a62-bfd7-3d58ddb6cf98\" (UID: \"9726c116-9266-4a62-bfd7-3d58ddb6cf98\") "
Nov 02 01:48:04.331889 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:04.331840    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9726c116-9266-4a62-bfd7-3d58ddb6cf98-kube-api-access-vgtn8\" (UniqueName: \"kubernetes.io/projected/9726c116-9266-4a62-bfd7-3d58ddb6cf98-kube-api-access-vgtn8\") pod \"9726c116-9266-4a62-bfd7-3d58ddb6cf98\" (UID: \"9726c116-9266-4a62-bfd7-3d58ddb6cf98\") "
Nov 02 01:48:04.332175 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:04.332139    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9726c116-9266-4a62-bfd7-3d58ddb6cf98-maas-api-openshift-ca" pod "9726c116-9266-4a62-bfd7-3d58ddb6cf98" (UID: "9726c116-9266-4a62-bfd7-3d58ddb6cf98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:04.332381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:04.332363    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f96ea7a-c7f4-429c-9c49-39a99091044f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f1b79b0-5q5hw\" (UID: \"2f96ea7a-c7f4-429c-9c49-39a99091044f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1b79b0-5q5hw"
Nov 02 01:48:04.332463 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:04.332409    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f96ea7a-c7f4-429c-9c49-39a99091044f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f1b79b0-5q5hw\" (UID: \"2f96ea7a-c7f4-429c-9c49-39a99091044f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1b79b0-5q5hw"
Nov 02 01:48:04.332684 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:04.332665    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt8bh\" (UniqueName: \"kubernetes.io/projected/2f96ea7a-c7f4-429c-9c49-39a99091044f-kube-api-access-mt8bh\") pod \"maas-api-revoke-sub-6f1b79b0-5q5hw\" (UID: \"2f96ea7a-c7f4-429c-9c49-39a99091044f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1b79b0-5q5hw"
Nov 02 01:48:04.332823 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:04.332802    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9726c116-9266-4a62-bfd7-3d58ddb6cf98-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:04.333224 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:04.333202    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f96ea7a-c7f4-429c-9c49-39a99091044f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f1b79b0-5q5hw\" (UID: \"2f96ea7a-c7f4-429c-9c49-39a99091044f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1b79b0-5q5hw"
Nov 02 01:48:04.334197 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:04.334169    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9726c116-9266-4a62-bfd7-3d58ddb6cf98-maas-api-xks-ca" pod "9726c116-9266-4a62-bfd7-3d58ddb6cf98" (UID: "9726c116-9266-4a62-bfd7-3d58ddb6cf98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:04.334318 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:04.334232    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9726c116-9266-4a62-bfd7-3d58ddb6cf98-kube-api-access-vgtn8" pod "9726c116-9266-4a62-bfd7-3d58ddb6cf98" (UID: "9726c116-9266-4a62-bfd7-3d58ddb6cf98"). InnerVolumeSpecName "kube-api-access-vgtn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:04.334792 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:04.334770    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f96ea7a-c7f4-429c-9c49-39a99091044f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f1b79b0-5q5hw\" (UID: \"2f96ea7a-c7f4-429c-9c49-39a99091044f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1b79b0-5q5hw"
Nov 02 01:48:04.396900 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:04.396864    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt8bh\" (UniqueName: \"kubernetes.io/projected/2f96ea7a-c7f4-429c-9c49-39a99091044f-kube-api-access-mt8bh\") pod \"maas-api-revoke-sub-6f1b79b0-5q5hw\" (UID: \"2f96ea7a-c7f4-429c-9c49-39a99091044f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1b79b0-5q5hw"
Nov 02 01:48:04.434894 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:04.434859    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9726c116-9266-4a62-bfd7-3d58ddb6cf98-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:04.435083 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:04.434899    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgtn8\" (UniqueName: \"kubernetes.io/projected/9726c116-9266-4a62-bfd7-3d58ddb6cf98-kube-api-access-vgtn8\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:04.475118 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:04.475089    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1b79b0-5q5hw"
Nov 02 01:48:04.853925 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:48:04.853882    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f96ea7a_c7f4_429c_9c49_39a99091044f.slice/crio-6a5385df06ba3d3beb5ef92925210382208c287d654de378e698f3af486e79ce WatchSource:0}: Error finding container 6a5385df06ba3d3beb5ef92925210382208c287d654de378e698f3af486e79ce: Status 404 returned error can't find the container with id 6a5385df06ba3d3beb5ef92925210382208c287d654de378e698f3af486e79ce
Nov 02 01:48:04.854487 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:04.854462    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f1b79b0-5q5hw"]
Nov 02 01:48:05.082040 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:05.082012    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1b79b0-5q5hw" event={"ID":"2f96ea7a-c7f4-429c-9c49-39a99091044f","Type":"ContainerStarted","Data":"6a5385df06ba3d3beb5ef92925210382208c287d654de378e698f3af486e79ce"}
Nov 02 01:48:05.083473 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:05.083436    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ed0482-2mkqb" event={"ID":"9726c116-9266-4a62-bfd7-3d58ddb6cf98","Type":"ContainerDied","Data":"1f37004020297bb25776c46261c94ed312801ca17843a4a167638637501858f9"}
Nov 02 01:48:05.083473 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:05.083461    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ed0482-2mkqb"
Nov 02 01:48:05.083622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:05.083479    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f37004020297bb25776c46261c94ed312801ca17843a4a167638637501858f9"
Nov 02 01:48:06.092978 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:06.092940    5639 generic.go:356] "Generic (PLEG): container finished" podID="2f96ea7a-c7f4-429c-9c49-39a99091044f" containerID="bf360c2b5bb56c9787c73e732b381dcf7d444afd0840f5a0c5c1c87f1d008535" exitCode=0
Nov 02 01:48:06.093440 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:06.093008    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1b79b0-5q5hw" event={"ID":"2f96ea7a-c7f4-429c-9c49-39a99091044f","Type":"ContainerDied","Data":"bf360c2b5bb56c9787c73e732b381dcf7d444afd0840f5a0c5c1c87f1d008535"}
Nov 02 01:48:07.252551 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:07.252530    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1b79b0-5q5hw"
Nov 02 01:48:07.376063 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:07.375982    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f96ea7a-c7f4-429c-9c49-39a99091044f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f96ea7a-c7f4-429c-9c49-39a99091044f-maas-api-openshift-ca\") pod \"2f96ea7a-c7f4-429c-9c49-39a99091044f\" (UID: \"2f96ea7a-c7f4-429c-9c49-39a99091044f\") "
Nov 02 01:48:07.376063 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:07.376017    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f96ea7a-c7f4-429c-9c49-39a99091044f-kube-api-access-mt8bh\" (UniqueName: \"kubernetes.io/projected/2f96ea7a-c7f4-429c-9c49-39a99091044f-kube-api-access-mt8bh\") pod \"2f96ea7a-c7f4-429c-9c49-39a99091044f\" (UID: \"2f96ea7a-c7f4-429c-9c49-39a99091044f\") "
Nov 02 01:48:07.376310 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:07.376292    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f96ea7a-c7f4-429c-9c49-39a99091044f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f96ea7a-c7f4-429c-9c49-39a99091044f-maas-api-xks-ca\") pod \"2f96ea7a-c7f4-429c-9c49-39a99091044f\" (UID: \"2f96ea7a-c7f4-429c-9c49-39a99091044f\") "
Nov 02 01:48:07.376492 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:07.376447    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f96ea7a-c7f4-429c-9c49-39a99091044f-maas-api-openshift-ca" pod "2f96ea7a-c7f4-429c-9c49-39a99091044f" (UID: "2f96ea7a-c7f4-429c-9c49-39a99091044f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:07.376900 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:07.376860    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f96ea7a-c7f4-429c-9c49-39a99091044f-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:07.380315 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:07.378725    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f96ea7a-c7f4-429c-9c49-39a99091044f-maas-api-xks-ca" pod "2f96ea7a-c7f4-429c-9c49-39a99091044f" (UID: "2f96ea7a-c7f4-429c-9c49-39a99091044f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:07.380315 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:07.378730    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f96ea7a-c7f4-429c-9c49-39a99091044f-kube-api-access-mt8bh" pod "2f96ea7a-c7f4-429c-9c49-39a99091044f" (UID: "2f96ea7a-c7f4-429c-9c49-39a99091044f"). InnerVolumeSpecName "kube-api-access-mt8bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:07.478822 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:07.478792    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f96ea7a-c7f4-429c-9c49-39a99091044f-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:07.478822 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:07.478819    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt8bh\" (UniqueName: \"kubernetes.io/projected/2f96ea7a-c7f4-429c-9c49-39a99091044f-kube-api-access-mt8bh\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:08.050117 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:08.050086    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8edfdd95-qjp8v"]
Nov 02 01:48:08.100040 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:08.100009    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8edfdd95-qjp8v"]
Nov 02 01:48:08.119518 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:08.119486    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1b79b0-5q5hw"
Nov 02 01:48:08.119700 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:08.119529    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1b79b0-5q5hw" event={"ID":"2f96ea7a-c7f4-429c-9c49-39a99091044f","Type":"ContainerDied","Data":"6a5385df06ba3d3beb5ef92925210382208c287d654de378e698f3af486e79ce"}
Nov 02 01:48:08.119700 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:08.119622    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a5385df06ba3d3beb5ef92925210382208c287d654de378e698f3af486e79ce"
Nov 02 01:48:09.219347 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:09.219314    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b4600f4-8f54-4755-815d-dd4e041065a3" path="/var/lib/kubelet/pods/6b4600f4-8f54-4755-815d-dd4e041065a3/volumes"
Nov 02 01:48:09.981784 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:09.981751    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"]
Nov 02 01:48:09.982644 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:09.982624    5639 cpu_manager.go:414] "containerMap: removing container" podUID="9726c116-9266-4a62-bfd7-3d58ddb6cf98" containerName="revoke-keys"
Nov 02 01:48:09.982720 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:09.982644    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9726c116-9266-4a62-bfd7-3d58ddb6cf98" containerName="revoke-keys"
Nov 02 01:48:09.982720 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:09.982678    5639 cpu_manager.go:414] "containerMap: removing container" podUID="2f96ea7a-c7f4-429c-9c49-39a99091044f" containerName="revoke-keys"
Nov 02 01:48:09.982720 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:09.982683    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f96ea7a-c7f4-429c-9c49-39a99091044f" containerName="revoke-keys"
Nov 02 01:48:09.982820 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:09.982812    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="9726c116-9266-4a62-bfd7-3d58ddb6cf98" containerName="revoke-keys"
Nov 02 01:48:09.982853 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:09.982822    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f96ea7a-c7f4-429c-9c49-39a99091044f" containerName="revoke-keys"
Nov 02 01:48:09.987655 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:09.987637    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.094392 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.094355    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"]
Nov 02 01:48:10.115610 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.115574    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.115610 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.115615    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.115852 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.115765    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97tj9\" (UniqueName: \"kubernetes.io/projected/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-kube-api-access-97tj9\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.115852 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.115824    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.115929 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.115884    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.116027 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.116005    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.116106 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.116090    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.116144 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.116125    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.116177 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.116164    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.217600 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.217566    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.217600 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.217599    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.217799 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.217631    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.217799 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.217697    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.217799 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.217711    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.217799 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.217764    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97tj9\" (UniqueName: \"kubernetes.io/projected/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-kube-api-access-97tj9\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.217799 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.217793    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.218058 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.217815    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.218058 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.217911    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.218058 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.218006    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.218058 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.218044    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.218383 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.218339    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.218457 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.218410    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.218630 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.218612    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.220074 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.220055    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.220378 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.220317    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.278860 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.278782    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.279001 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.278887    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97tj9\" (UniqueName: \"kubernetes.io/projected/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-kube-api-access-97tj9\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.302546 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.302512    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:10.419182 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.419143    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"]
Nov 02 01:48:10.426471 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.426447    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.484643 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.484617    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"]
Nov 02 01:48:10.522651 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.522614    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8596379-fe63-4152-8569-1b16a8f75d70-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.522651 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.522657    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.522897 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.522675    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d29zf\" (UniqueName: \"kubernetes.io/projected/a8596379-fe63-4152-8569-1b16a8f75d70-kube-api-access-d29zf\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.522897 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.522784    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.522897 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.522813    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8596379-fe63-4152-8569-1b16a8f75d70-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.522897 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.522837    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.523112 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.522951    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8596379-fe63-4152-8569-1b16a8f75d70-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.523112 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.523028    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.523215 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.523119    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.624024 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.623991    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.624217 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.624056    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.624217 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.624102    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8596379-fe63-4152-8569-1b16a8f75d70-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.624217 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.624141    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.624217 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.624164    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d29zf\" (UniqueName: \"kubernetes.io/projected/a8596379-fe63-4152-8569-1b16a8f75d70-kube-api-access-d29zf\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.624217 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.624214    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.624531 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.624253    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8596379-fe63-4152-8569-1b16a8f75d70-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.624531 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.624323    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.624531 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.624431    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8596379-fe63-4152-8569-1b16a8f75d70-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.624531 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.624470    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.624531 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.624504    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.625137 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.624557    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.625137 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.624810    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.625137 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.625027    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8596379-fe63-4152-8569-1b16a8f75d70-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.626781 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.626761    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.626882 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.626802    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8596379-fe63-4152-8569-1b16a8f75d70-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.688728 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.688689    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d29zf\" (UniqueName: \"kubernetes.io/projected/a8596379-fe63-4152-8569-1b16a8f75d70-kube-api-access-d29zf\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.688908 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.688852    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8596379-fe63-4152-8569-1b16a8f75d70-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.715854 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.715822    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"]
Nov 02 01:48:10.722369 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.722348    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.726524 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.726501    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.726640 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.726558    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.726640 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.726623    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.726715 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.726667    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f799283-3131-46a5-b010-628029210780-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.726840 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.726816    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.726939 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.726919    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f799283-3131-46a5-b010-628029210780-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.727030 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.727013    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f799283-3131-46a5-b010-628029210780-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.727089 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.727076    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.727175 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.727162    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j88k\" (UniqueName: \"kubernetes.io/projected/1f799283-3131-46a5-b010-628029210780-kube-api-access-5j88k\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.741299 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.741252    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"]
Nov 02 01:48:10.741417 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:48:10.741249    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b58fbf_2abd_4e6e_b40c_f7dcd8006713.slice/crio-b56480489d81ce319b12c9233526ed737d7036183fd0ab683ec6d38e5c1e54e9 WatchSource:0}: Error finding container b56480489d81ce319b12c9233526ed737d7036183fd0ab683ec6d38e5c1e54e9: Status 404 returned error can't find the container with id b56480489d81ce319b12c9233526ed737d7036183fd0ab683ec6d38e5c1e54e9
Nov 02 01:48:10.743921 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.743890    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:48:10.744005 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.743950    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:48:10.744005 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.743979    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:48:10.745940 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.745918    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:10.802159 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.802034    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"]
Nov 02 01:48:10.830693 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.828108    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f799283-3131-46a5-b010-628029210780-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.830693 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.828181    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f799283-3131-46a5-b010-628029210780-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.830693 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.828241    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.830693 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.828340    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j88k\" (UniqueName: \"kubernetes.io/projected/1f799283-3131-46a5-b010-628029210780-kube-api-access-5j88k\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.830693 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.828388    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.830693 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.828447    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.830693 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.828483    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.830693 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.828548    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f799283-3131-46a5-b010-628029210780-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.830693 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.828630    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.830693 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.828948    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.830693 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.829254    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.830693 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.829427    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.830693 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.829724    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.830693 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.830360    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f799283-3131-46a5-b010-628029210780-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.832586 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.832561    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.833399 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.833175    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f799283-3131-46a5-b010-628029210780-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.895717 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.895633    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f799283-3131-46a5-b010-628029210780-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.896117 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.896095    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j88k\" (UniqueName: \"kubernetes.io/projected/1f799283-3131-46a5-b010-628029210780-kube-api-access-5j88k\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz\" (UID: \"1f799283-3131-46a5-b010-628029210780\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:10.998042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:10.998009    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"]
Nov 02 01:48:11.003873 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.003851    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.038555 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.038520    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:11.056306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.055210    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"]
Nov 02 01:48:11.134080 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.134047    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.134213 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.134127    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.134213 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.134164    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.134361 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.134300    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.134423 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.134376    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.134481 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.134454    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.134532 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.134493    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.134581 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.134527    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvqkf\" (UniqueName: \"kubernetes.io/projected/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-kube-api-access-zvqkf\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.134645 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.134592    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.155978 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.155891    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f" event={"ID":"94b58fbf-2abd-4e6e-b40c-f7dcd8006713","Type":"ContainerStarted","Data":"eabb6763d3cfe188665b9843bda64fd0788154b7f7b0cbb5e7981705f310d023"}
Nov 02 01:48:11.155978 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.155967    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f" event={"ID":"94b58fbf-2abd-4e6e-b40c-f7dcd8006713","Type":"ContainerStarted","Data":"b56480489d81ce319b12c9233526ed737d7036183fd0ab683ec6d38e5c1e54e9"}
Nov 02 01:48:11.177971 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.177941    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c6366e-fs5gv"]
Nov 02 01:48:11.183817 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.183795    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c6366e-fs5gv"
Nov 02 01:48:11.204353 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.204326    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"]
Nov 02 01:48:11.204803 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:48:11.204774    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8596379_fe63_4152_8569_1b16a8f75d70.slice/crio-cf588bc4b324eafa0716d2b55cdd067b043dae4aaf0be3ed0039f04fa36a9e53 WatchSource:0}: Error finding container cf588bc4b324eafa0716d2b55cdd067b043dae4aaf0be3ed0039f04fa36a9e53: Status 404 returned error can't find the container with id cf588bc4b324eafa0716d2b55cdd067b043dae4aaf0be3ed0039f04fa36a9e53
Nov 02 01:48:11.208207 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.208178    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:48:11.208312 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.208251    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:48:11.208391 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.208317    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:48:11.239195 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.239167    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c6366e-fs5gv"]
Nov 02 01:48:11.239671 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.239457    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:48:11.239671 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.239514    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:48:11.239671 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.239591    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:48:11.239671 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.239649    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:48:11.240358 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.240339    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.240459 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.240394    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be73d62-1a59-4660-9515-aad1c170fb86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59c6366e-fs5gv\" (UID: \"8be73d62-1a59-4660-9515-aad1c170fb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c6366e-fs5gv"
Nov 02 01:48:11.240459 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.240425    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.240459 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.240446    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.240654 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.240469    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be73d62-1a59-4660-9515-aad1c170fb86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59c6366e-fs5gv\" (UID: \"8be73d62-1a59-4660-9515-aad1c170fb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c6366e-fs5gv"
Nov 02 01:48:11.240654 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.240544    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.240654 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.240566    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5dnx\" (UniqueName: \"kubernetes.io/projected/8be73d62-1a59-4660-9515-aad1c170fb86-kube-api-access-r5dnx\") pod \"maas-api-revoke-sub-59c6366e-fs5gv\" (UID: \"8be73d62-1a59-4660-9515-aad1c170fb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c6366e-fs5gv"
Nov 02 01:48:11.240654 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.240597    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.240654 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.240653    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.240908 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.240678    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.240908 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.240697    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvqkf\" (UniqueName: \"kubernetes.io/projected/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-kube-api-access-zvqkf\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.240908 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.240739    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.243098 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.241489    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.243098 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.242202    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.243098 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.242485    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.243912 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.243885    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.244443 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.244421    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.248352 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.247992    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.249442 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.249363    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.281783 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.281646    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f" podStartSLOduration=2.281630019 podStartE2EDuration="2.281630019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:11.276382258 +0000 UTC m=+1356.180064406" watchObservedRunningTime="2026-11-02 01:48:11.281630019 +0000 UTC m=+1356.185312131"
Nov 02 01:48:11.302586 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.302550    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.302758 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.302739    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:11.304684 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.304378    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 01:48:11.304684 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.304429    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f" podUID="94b58fbf-2abd-4e6e-b40c-f7dcd8006713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 01:48:11.305738 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.305652    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvqkf\" (UniqueName: \"kubernetes.io/projected/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-kube-api-access-zvqkf\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.319539 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.319516    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:11.344095 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.342369    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be73d62-1a59-4660-9515-aad1c170fb86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59c6366e-fs5gv\" (UID: \"8be73d62-1a59-4660-9515-aad1c170fb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c6366e-fs5gv"
Nov 02 01:48:11.344095 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.342444    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be73d62-1a59-4660-9515-aad1c170fb86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59c6366e-fs5gv\" (UID: \"8be73d62-1a59-4660-9515-aad1c170fb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c6366e-fs5gv"
Nov 02 01:48:11.344095 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.342545    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5dnx\" (UniqueName: \"kubernetes.io/projected/8be73d62-1a59-4660-9515-aad1c170fb86-kube-api-access-r5dnx\") pod \"maas-api-revoke-sub-59c6366e-fs5gv\" (UID: \"8be73d62-1a59-4660-9515-aad1c170fb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c6366e-fs5gv"
Nov 02 01:48:11.344095 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.343755    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be73d62-1a59-4660-9515-aad1c170fb86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59c6366e-fs5gv\" (UID: \"8be73d62-1a59-4660-9515-aad1c170fb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c6366e-fs5gv"
Nov 02 01:48:11.346702 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.346675    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be73d62-1a59-4660-9515-aad1c170fb86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59c6366e-fs5gv\" (UID: \"8be73d62-1a59-4660-9515-aad1c170fb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c6366e-fs5gv"
Nov 02 01:48:11.405974 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.405793    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5dnx\" (UniqueName: \"kubernetes.io/projected/8be73d62-1a59-4660-9515-aad1c170fb86-kube-api-access-r5dnx\") pod \"maas-api-revoke-sub-59c6366e-fs5gv\" (UID: \"8be73d62-1a59-4660-9515-aad1c170fb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c6366e-fs5gv"
Nov 02 01:48:11.469601 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.469514    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"]
Nov 02 01:48:11.507728 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.507658    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c6366e-fs5gv"
Nov 02 01:48:11.511696 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:48:11.511671    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f799283_3131_46a5_b010_628029210780.slice/crio-aeac582c7bb9ddb6a5835911f771b1c643c56fe99155bbd47dc40384e72735fc WatchSource:0}: Error finding container aeac582c7bb9ddb6a5835911f771b1c643c56fe99155bbd47dc40384e72735fc: Status 404 returned error can't find the container with id aeac582c7bb9ddb6a5835911f771b1c643c56fe99155bbd47dc40384e72735fc
Nov 02 01:48:11.514810 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.514772    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:48:11.514926 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.514860    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:48:11.514926 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.514908    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:48:11.750052 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.749842    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"]
Nov 02 01:48:11.752546 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:48:11.752517    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e35e13f_b6eb_4a4c_8142_f54a5c621a97.slice/crio-26414b2a462bd223de39f38b5ae0b0cedd89a3d4d55df044bfe1c9fc5ecbb194 WatchSource:0}: Error finding container 26414b2a462bd223de39f38b5ae0b0cedd89a3d4d55df044bfe1c9fc5ecbb194: Status 404 returned error can't find the container with id 26414b2a462bd223de39f38b5ae0b0cedd89a3d4d55df044bfe1c9fc5ecbb194
Nov 02 01:48:11.757243 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.757199    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:48:11.757395 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.757314    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:48:11.757395 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.757363    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:48:11.826364 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.826316    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76c31dd2-xt2x4"]
Nov 02 01:48:11.832657 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.832633    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c31dd2-xt2x4"
Nov 02 01:48:11.856520 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.856074    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f02f9c71-728b-49af-9657-c3574c42f998-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76c31dd2-xt2x4\" (UID: \"f02f9c71-728b-49af-9657-c3574c42f998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c31dd2-xt2x4"
Nov 02 01:48:11.856520 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.856211    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fkj7\" (UniqueName: \"kubernetes.io/projected/f02f9c71-728b-49af-9657-c3574c42f998-kube-api-access-2fkj7\") pod \"maas-api-revoke-sub-76c31dd2-xt2x4\" (UID: \"f02f9c71-728b-49af-9657-c3574c42f998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c31dd2-xt2x4"
Nov 02 01:48:11.856520 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.856383    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f02f9c71-728b-49af-9657-c3574c42f998-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76c31dd2-xt2x4\" (UID: \"f02f9c71-728b-49af-9657-c3574c42f998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c31dd2-xt2x4"
Nov 02 01:48:11.904828 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:11.904801    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76c31dd2-xt2x4"]
Nov 02 01:48:12.020359 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:12.019745    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f02f9c71-728b-49af-9657-c3574c42f998-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76c31dd2-xt2x4\" (UID: \"f02f9c71-728b-49af-9657-c3574c42f998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c31dd2-xt2x4"
Nov 02 01:48:12.020359 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:12.019893    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fkj7\" (UniqueName: \"kubernetes.io/projected/f02f9c71-728b-49af-9657-c3574c42f998-kube-api-access-2fkj7\") pod \"maas-api-revoke-sub-76c31dd2-xt2x4\" (UID: \"f02f9c71-728b-49af-9657-c3574c42f998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c31dd2-xt2x4"
Nov 02 01:48:12.020359 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:12.020075    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f02f9c71-728b-49af-9657-c3574c42f998-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76c31dd2-xt2x4\" (UID: \"f02f9c71-728b-49af-9657-c3574c42f998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c31dd2-xt2x4"
Nov 02 01:48:12.045339 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:12.020904    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f02f9c71-728b-49af-9657-c3574c42f998-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76c31dd2-xt2x4\" (UID: \"f02f9c71-728b-49af-9657-c3574c42f998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c31dd2-xt2x4"
Nov 02 01:48:12.106220 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:12.106176    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c6366e-fs5gv"]
Nov 02 01:48:12.113183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:12.112680    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fkj7\" (UniqueName: \"kubernetes.io/projected/f02f9c71-728b-49af-9657-c3574c42f998-kube-api-access-2fkj7\") pod \"maas-api-revoke-sub-76c31dd2-xt2x4\" (UID: \"f02f9c71-728b-49af-9657-c3574c42f998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c31dd2-xt2x4"
Nov 02 01:48:12.141666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:12.141634    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f02f9c71-728b-49af-9657-c3574c42f998-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76c31dd2-xt2x4\" (UID: \"f02f9c71-728b-49af-9657-c3574c42f998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c31dd2-xt2x4"
Nov 02 01:48:12.223240 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:12.222864    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c31dd2-xt2x4"
Nov 02 01:48:12.230419 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:12.230346    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c6366e-fs5gv" event={"ID":"8be73d62-1a59-4660-9515-aad1c170fb86","Type":"ContainerStarted","Data":"6eb0dbc07eb6dad987301ea6e74d48f0d02035a28651af9c2f733c82cccfe38e"}
Nov 02 01:48:12.238250 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:12.238218    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4" event={"ID":"a8596379-fe63-4152-8569-1b16a8f75d70","Type":"ContainerStarted","Data":"5b268a42d4f06300441305b3b0bdf09472821369b44baf84a48ba67ba57254e9"}
Nov 02 01:48:12.238430 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:12.238299    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4" event={"ID":"a8596379-fe63-4152-8569-1b16a8f75d70","Type":"ContainerStarted","Data":"cf588bc4b324eafa0716d2b55cdd067b043dae4aaf0be3ed0039f04fa36a9e53"}
Nov 02 01:48:12.241170 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:12.241135    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz" event={"ID":"1f799283-3131-46a5-b010-628029210780","Type":"ContainerStarted","Data":"19b8c99f9f3e6a460f55fbc5880b434a658cea21501e8c09be3c18e9d42e1b2a"}
Nov 02 01:48:12.241699 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:12.241198    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz" event={"ID":"1f799283-3131-46a5-b010-628029210780","Type":"ContainerStarted","Data":"aeac582c7bb9ddb6a5835911f771b1c643c56fe99155bbd47dc40384e72735fc"}
Nov 02 01:48:12.243149 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:12.243125    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r" event={"ID":"4e35e13f-b6eb-4a4c-8142-f54a5c621a97","Type":"ContainerStarted","Data":"9dc87c280ab43af3fd72d7b2532fcbc0505d9fc0addfd1a3bab0e42853db8e3a"}
Nov 02 01:48:12.243244 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:12.243175    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r" event={"ID":"4e35e13f-b6eb-4a4c-8142-f54a5c621a97","Type":"ContainerStarted","Data":"26414b2a462bd223de39f38b5ae0b0cedd89a3d4d55df044bfe1c9fc5ecbb194"}
Nov 02 01:48:12.303301 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:12.302812    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 01:48:12.303301 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:12.302867    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f" podUID="94b58fbf-2abd-4e6e-b40c-f7dcd8006713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 01:48:12.322778 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:12.322696    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:12.328567 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:12.328415    5639 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 02 01:48:12.328567 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:12.328487    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r" podUID="4e35e13f-b6eb-4a4c-8142-f54a5c621a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 02 01:48:12.528424 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:12.513852    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4" podStartSLOduration=2.513831122 podStartE2EDuration="2.513831122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:12.377037184 +0000 UTC m=+1357.280719298" watchObservedRunningTime="2026-11-02 01:48:12.513831122 +0000 UTC m=+1357.417513235"
Nov 02 01:48:12.550648 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:12.549510    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r" podStartSLOduration=2.549491409 podStartE2EDuration="2.549491409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:12.512163154 +0000 UTC m=+1357.415845278" watchObservedRunningTime="2026-11-02 01:48:12.549491409 +0000 UTC m=+1357.453173522"
Nov 02 01:48:12.793304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:12.752094    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:12.793304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:12.754716    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz" podStartSLOduration=2.75469733 podStartE2EDuration="2.75469733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:12.752504349 +0000 UTC m=+1357.656186475" watchObservedRunningTime="2026-11-02 01:48:12.75469733 +0000 UTC m=+1357.658379447"
Nov 02 01:48:12.871316 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:12.867728    5639 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 02 01:48:12.871316 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:12.867785    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4" podUID="a8596379-fe63-4152-8569-1b16a8f75d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 02 01:48:13.010384 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:13.000625    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76c31dd2-xt2x4"]
Nov 02 01:48:13.051303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:13.047938    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:13.082316 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:48:13.077425    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02f9c71_728b_49af_9657_c3574c42f998.slice/crio-4889d2d51826bf33555f1532254c6ef27b372674761f8075f5305e0160674e57 WatchSource:0}: Error finding container 4889d2d51826bf33555f1532254c6ef27b372674761f8075f5305e0160674e57: Status 404 returned error can't find the container with id 4889d2d51826bf33555f1532254c6ef27b372674761f8075f5305e0160674e57
Nov 02 01:48:13.082316 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:13.078466    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 02 01:48:13.082316 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:13.078511    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz" podUID="1f799283-3131-46a5-b010-628029210780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 02 01:48:13.491559 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:13.446394    5639 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 02 01:48:13.491559 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:13.479555    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r" podUID="4e35e13f-b6eb-4a4c-8142-f54a5c621a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 02 01:48:13.491559 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:13.480584    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 01:48:13.491559 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:13.480683    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f" podUID="94b58fbf-2abd-4e6e-b40c-f7dcd8006713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 01:48:13.699398 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:13.696395    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:13.762356 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:13.758178    5639 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 02 01:48:13.799322 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:13.794397    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4" podUID="a8596379-fe63-4152-8569-1b16a8f75d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 02 01:48:13.799322 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:13.757774    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:13.801398 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:13.801360    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c6366e-fs5gv" event={"ID":"8be73d62-1a59-4660-9515-aad1c170fb86","Type":"ContainerStarted","Data":"cbc6f984ed172f9e9fe6d0e992bde1a1629874dbefa7ad9553e94e3d5d30d816"}
Nov 02 01:48:13.830474 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:13.813442    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c31dd2-xt2x4" event={"ID":"f02f9c71-728b-49af-9657-c3574c42f998","Type":"ContainerStarted","Data":"4889d2d51826bf33555f1532254c6ef27b372674761f8075f5305e0160674e57"}
Nov 02 01:48:13.874608 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:13.874559    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:13.902311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:13.880718    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:13.980435 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:13.977629    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:14.003687 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:14.003635    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:14.061887 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:14.042807    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 02 01:48:14.061887 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:14.042913    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz" podUID="1f799283-3131-46a5-b010-628029210780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 02 01:48:14.061887 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:14.056431    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": context deadline exceeded"
Nov 02 01:48:14.153379 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:14.139384    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:14.303063 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:14.302985    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 01:48:14.303063 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:14.303037    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f" podUID="94b58fbf-2abd-4e6e-b40c-f7dcd8006713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 01:48:14.321322 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:14.319886    5639 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 02 01:48:14.321322 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:14.319947    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r" podUID="4e35e13f-b6eb-4a4c-8142-f54a5c621a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 02 01:48:14.469302 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:14.468489    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:14.619416 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:14.619371    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:14.701315 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:14.700366    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:14.701315 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:14.700809    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:48:14.799307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:14.765828    5639 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 02 01:48:14.799620 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:14.799584    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4" podUID="a8596379-fe63-4152-8569-1b16a8f75d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 02 01:48:14.799818 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:14.795644    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:14.800324 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:14.800301    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:48:14.904363 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:14.903336    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:14.904363 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:14.903699    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:48:14.938311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:14.904664    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:48:14.938311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:14.904724    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:48:14.992571 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:14.992529    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:14.993223 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:14.993199    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:48:15.065415 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:15.015232    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:15.107504 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:15.076352    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 02 01:48:15.132519 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:15.130415    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz" podUID="1f799283-3131-46a5-b010-628029210780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 02 01:48:15.132519 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:15.130569    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:48:15.244362 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:15.241428    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:15.573405 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:15.375830    5639 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 02 01:48:15.573405 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:15.375893    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r" podUID="4e35e13f-b6eb-4a4c-8142-f54a5c621a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 02 01:48:15.573405 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:15.375986    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 01:48:15.573405 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:15.376008    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f" podUID="94b58fbf-2abd-4e6e-b40c-f7dcd8006713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 01:48:15.573405 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:15.484518    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:15.573405 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:15.486387    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:48:15.641438 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:15.641372    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:15.712439 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:15.712397    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:15.832317 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:15.831446    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:15.916429 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:15.916378    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:48:15.953413 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:15.931428    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 02 01:48:16.007306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:15.997397    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:16.118938 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:16.118847    5639 generic.go:356] "Generic (PLEG): container finished" podID="8be73d62-1a59-4660-9515-aad1c170fb86" containerID="cbc6f984ed172f9e9fe6d0e992bde1a1629874dbefa7ad9553e94e3d5d30d816" exitCode=0
Nov 02 01:48:16.119342 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:16.119308    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:16.248304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:16.244447    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:48:16.433464 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:16.433367    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:16.498472 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:16.498421    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:16.499196 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:16.499168    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:48:16.499411 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:16.499390    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c31dd2-xt2x4" event={"ID":"f02f9c71-728b-49af-9657-c3574c42f998","Type":"ContainerStarted","Data":"0dc16592262cc54e5da0c9c56113317f431f5f6633e6478f68237bd3e71d0f37"}
Nov 02 01:48:16.549303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:16.537589    5639 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 02 01:48:16.554311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:16.550499    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4" podUID="a8596379-fe63-4152-8569-1b16a8f75d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 02 01:48:16.618317 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:16.590355    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:48:16.618317 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:16.590439    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c6366e-fs5gv" event={"ID":"8be73d62-1a59-4660-9515-aad1c170fb86","Type":"ContainerDied","Data":"cbc6f984ed172f9e9fe6d0e992bde1a1629874dbefa7ad9553e94e3d5d30d816"}
Nov 02 01:48:16.651303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:16.634513    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 02 01:48:16.651303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:16.634582    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz" podUID="1f799283-3131-46a5-b010-628029210780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 02 01:48:16.673348 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:16.657820    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:16.673348 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:16.658213    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 01:48:16.673348 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:16.658267    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f" podUID="94b58fbf-2abd-4e6e-b40c-f7dcd8006713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 01:48:16.673348 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:16.658390    5639 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 02 01:48:16.673348 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:16.658412    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r" podUID="4e35e13f-b6eb-4a4c-8142-f54a5c621a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 02 01:48:17.169668 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:17.169609    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 02 01:48:17.169922 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:17.169893    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz" podUID="1f799283-3131-46a5-b010-628029210780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 02 01:48:17.520381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:17.502802    5639 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 02 01:48:17.520381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:17.502865    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r" podUID="4e35e13f-b6eb-4a4c-8142-f54a5c621a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 02 01:48:17.561377 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:17.550361    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:17.593631 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:17.552790    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:17.593882 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:17.593798    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:17.617472 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:17.562228    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:48:17.764497 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:17.562362    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:17.765846 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:17.562414    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 02 01:48:17.766687 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:17.562441    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:17.798896 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:17.654211    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 01:48:17.798896 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:17.793754    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f" podUID="94b58fbf-2abd-4e6e-b40c-f7dcd8006713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 01:48:17.798896 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:17.562303    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:48:17.798896 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:17.654545    5639 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 02 01:48:17.798896 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:17.793955    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4" podUID="a8596379-fe63-4152-8569-1b16a8f75d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 02 01:48:17.798896 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:17.798150    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:17.980332 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:17.973040    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:18.020768 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:18.018468    5639 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 02 01:48:18.020768 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:18.018532    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4" podUID="a8596379-fe63-4152-8569-1b16a8f75d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 02 01:48:18.131551 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:18.131519    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 02 01:48:18.139514 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:18.139471    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz" podUID="1f799283-3131-46a5-b010-628029210780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 02 01:48:18.358849 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:18.352927    5639 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 02 01:48:18.358849 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:18.352986    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r" podUID="4e35e13f-b6eb-4a4c-8142-f54a5c621a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 02 01:48:18.393293 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:18.393237    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 01:48:18.401781 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:18.401734    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f" podUID="94b58fbf-2abd-4e6e-b40c-f7dcd8006713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 01:48:18.496305 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:48:18.475403    5639 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.114s"
Nov 02 01:48:18.497343 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:18.497315    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee317f08-6552p"]
Nov 02 01:48:18.599210 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:18.565838    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee317f08-6552p"]
Nov 02 01:48:18.599647 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:18.599610    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:18.627301 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:18.602083    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:48:18.666364 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:18.615601    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:48:18.666364 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:18.643597    5639 generic.go:356] "Generic (PLEG): container finished" podID="f02f9c71-728b-49af-9657-c3574c42f998" containerID="0dc16592262cc54e5da0c9c56113317f431f5f6633e6478f68237bd3e71d0f37" exitCode=0
Nov 02 01:48:18.666364 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:18.654543    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c31dd2-xt2x4" event={"ID":"f02f9c71-728b-49af-9657-c3574c42f998","Type":"ContainerDied","Data":"0dc16592262cc54e5da0c9c56113317f431f5f6633e6478f68237bd3e71d0f37"}
Nov 02 01:48:18.666364 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:18.663598    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:18.701704 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:18.696146    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:18.754333 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:18.747547    5639 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 02 01:48:18.754789 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:18.754757    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4" podUID="a8596379-fe63-4152-8569-1b16a8f75d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 02 01:48:18.880328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:18.879579    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 02 01:48:18.880328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:18.879964    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:18.947872 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:18.947771    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:18.949550 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:18.949520    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:19.040131 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.040097    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c6366e-fs5gv"
Nov 02 01:48:19.109196 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.109168    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:48:19.138359 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.138261    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:19.139936 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.139908    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 02 01:48:19.140183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.140159    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz" podUID="1f799283-3131-46a5-b010-628029210780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 02 01:48:19.437265 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.339713    5639 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 02 01:48:19.437552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.437524    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r" podUID="4e35e13f-b6eb-4a4c-8142-f54a5c621a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 02 01:48:19.439316 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.439265    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 01:48:19.439457 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.439334    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f" podUID="94b58fbf-2abd-4e6e-b40c-f7dcd8006713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 01:48:19.446720 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.446644    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c31dd2-xt2x4" podStartSLOduration=8.446625853 podStartE2EDuration="8.446625853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:19.013106325 +0000 UTC m=+1363.916788440" watchObservedRunningTime="2026-11-02 01:48:19.446625853 +0000 UTC m=+1364.350307969"
Nov 02 01:48:19.487382 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.465590    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8be73d62-1a59-4660-9515-aad1c170fb86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be73d62-1a59-4660-9515-aad1c170fb86-maas-api-xks-ca\") pod \"8be73d62-1a59-4660-9515-aad1c170fb86\" (UID: \"8be73d62-1a59-4660-9515-aad1c170fb86\") "
Nov 02 01:48:19.504616 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.504575    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2" path="/var/lib/kubelet/pods/d72a3c94-8d43-4a1f-b2f5-e74d2839a5f2/volumes"
Nov 02 01:48:19.509668 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.509640    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8be73d62-1a59-4660-9515-aad1c170fb86-kube-api-access-r5dnx\" (UniqueName: \"kubernetes.io/projected/8be73d62-1a59-4660-9515-aad1c170fb86-kube-api-access-r5dnx\") pod \"8be73d62-1a59-4660-9515-aad1c170fb86\" (UID: \"8be73d62-1a59-4660-9515-aad1c170fb86\") "
Nov 02 01:48:19.509948 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.509925    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8be73d62-1a59-4660-9515-aad1c170fb86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be73d62-1a59-4660-9515-aad1c170fb86-maas-api-openshift-ca\") pod \"8be73d62-1a59-4660-9515-aad1c170fb86\" (UID: \"8be73d62-1a59-4660-9515-aad1c170fb86\") "
Nov 02 01:48:19.519269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.518684    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be73d62-1a59-4660-9515-aad1c170fb86-maas-api-openshift-ca" pod "8be73d62-1a59-4660-9515-aad1c170fb86" (UID: "8be73d62-1a59-4660-9515-aad1c170fb86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:19.630313 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.629239    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be73d62-1a59-4660-9515-aad1c170fb86-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:19.672803 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.672764    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be73d62-1a59-4660-9515-aad1c170fb86-kube-api-access-r5dnx" pod "8be73d62-1a59-4660-9515-aad1c170fb86" (UID: "8be73d62-1a59-4660-9515-aad1c170fb86"). InnerVolumeSpecName "kube-api-access-r5dnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:19.706757 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.706632    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8be73d62-1a59-4660-9515-aad1c170fb86-maas-api-xks-ca" pod "8be73d62-1a59-4660-9515-aad1c170fb86" (UID: "8be73d62-1a59-4660-9515-aad1c170fb86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:19.707678 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.707652    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8233db32-47lt9"]
Nov 02 01:48:19.707862 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.707850    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8a6a89a-khrmb"]
Nov 02 01:48:19.708033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.708020    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:48:19.708180 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.708168    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:48:19.708394 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.708380    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:48:19.708568 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.708556    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:48:19.708675 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.708665    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8233db32-47lt9"]
Nov 02 01:48:19.708791 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.708781    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8a6a89a-khrmb"]
Nov 02 01:48:19.708947 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.708937    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:48:19.709160 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.709147    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:48:19.709332 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.709319    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:48:19.734192 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.731821    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5dnx\" (UniqueName: \"kubernetes.io/projected/8be73d62-1a59-4660-9515-aad1c170fb86-kube-api-access-r5dnx\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:19.734192 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.731860    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be73d62-1a59-4660-9515-aad1c170fb86-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:19.744669 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.744640    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c6366e-fs5gv"
Nov 02 01:48:19.746994 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.746956    5639 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 02 01:48:19.747153 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.747123    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4" podUID="a8596379-fe63-4152-8569-1b16a8f75d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 02 01:48:19.751721 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.747142    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c6366e-fs5gv" event={"ID":"8be73d62-1a59-4660-9515-aad1c170fb86","Type":"ContainerDied","Data":"6eb0dbc07eb6dad987301ea6e74d48f0d02035a28651af9c2f733c82cccfe38e"}
Nov 02 01:48:19.751869 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:19.751855    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6eb0dbc07eb6dad987301ea6e74d48f0d02035a28651af9c2f733c82cccfe38e"
Nov 02 01:48:20.042521 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:20.042482    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 02 01:48:20.042996 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:20.042543    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz" podUID="1f799283-3131-46a5-b010-628029210780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 02 01:48:20.540335 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:20.535217    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c31dd2-xt2x4"
Nov 02 01:48:20.540335 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:20.536052    5639 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 02 01:48:20.540335 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:20.536113    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r" podUID="4e35e13f-b6eb-4a4c-8142-f54a5c621a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 02 01:48:20.741696 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:20.736751    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:20.741696 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:20.737541    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:20.768235 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:20.768187    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:48:20.785616 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:20.785589    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c31dd2-xt2x4"
Nov 02 01:48:20.796986 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:20.796895    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c31dd2-xt2x4" event={"ID":"f02f9c71-728b-49af-9657-c3574c42f998","Type":"ContainerDied","Data":"4889d2d51826bf33555f1532254c6ef27b372674761f8075f5305e0160674e57"}
Nov 02 01:48:20.797232 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:20.797212    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4889d2d51826bf33555f1532254c6ef27b372674761f8075f5305e0160674e57"
Nov 02 01:48:20.966322 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:20.962239    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:20.966322 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:20.963543    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:20.966322 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:20.965853    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:48:21.007994 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.006137    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f02f9c71-728b-49af-9657-c3574c42f998-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f02f9c71-728b-49af-9657-c3574c42f998-maas-api-xks-ca\") pod \"f02f9c71-728b-49af-9657-c3574c42f998\" (UID: \"f02f9c71-728b-49af-9657-c3574c42f998\") "
Nov 02 01:48:21.007994 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.006207    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f02f9c71-728b-49af-9657-c3574c42f998-kube-api-access-2fkj7\" (UniqueName: \"kubernetes.io/projected/f02f9c71-728b-49af-9657-c3574c42f998-kube-api-access-2fkj7\") pod \"f02f9c71-728b-49af-9657-c3574c42f998\" (UID: \"f02f9c71-728b-49af-9657-c3574c42f998\") "
Nov 02 01:48:21.007994 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.006505    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f02f9c71-728b-49af-9657-c3574c42f998-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f02f9c71-728b-49af-9657-c3574c42f998-maas-api-openshift-ca\") pod \"f02f9c71-728b-49af-9657-c3574c42f998\" (UID: \"f02f9c71-728b-49af-9657-c3574c42f998\") "
Nov 02 01:48:21.012952 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.012909    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f02f9c71-728b-49af-9657-c3574c42f998-maas-api-openshift-ca" pod "f02f9c71-728b-49af-9657-c3574c42f998" (UID: "f02f9c71-728b-49af-9657-c3574c42f998"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:21.032566 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.032509    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02f9c71-728b-49af-9657-c3574c42f998-kube-api-access-2fkj7" pod "f02f9c71-728b-49af-9657-c3574c42f998" (UID: "f02f9c71-728b-49af-9657-c3574c42f998"). InnerVolumeSpecName "kube-api-access-2fkj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:21.033196 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.033153    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f02f9c71-728b-49af-9657-c3574c42f998-maas-api-xks-ca" pod "f02f9c71-728b-49af-9657-c3574c42f998" (UID: "f02f9c71-728b-49af-9657-c3574c42f998"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:21.113996 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.113298    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f02f9c71-728b-49af-9657-c3574c42f998-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:21.113996 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.113336    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fkj7\" (UniqueName: \"kubernetes.io/projected/f02f9c71-728b-49af-9657-c3574c42f998-kube-api-access-2fkj7\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:21.113996 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.113352    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f02f9c71-728b-49af-9657-c3574c42f998-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:21.283426 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.283389    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40c88d48-ec32-4b53-b6e7-96e02b531e82" path="/var/lib/kubelet/pods/40c88d48-ec32-4b53-b6e7-96e02b531e82/volumes"
Nov 02 01:48:21.284442 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.284384    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d77e8a8b-09b3-4122-9a17-2a1da2f5ab77" path="/var/lib/kubelet/pods/d77e8a8b-09b3-4122-9a17-2a1da2f5ab77/volumes"
Nov 02 01:48:21.540988 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.540920    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:21.544237 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.544201    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:21.545488 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.545444    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:48:21.645002 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.644961    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-372e00cd-kz2jm"]
Nov 02 01:48:21.648888 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.647988    5639 cpu_manager.go:414] "containerMap: removing container" podUID="8be73d62-1a59-4660-9515-aad1c170fb86" containerName="revoke-keys"
Nov 02 01:48:21.648888 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.648013    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8be73d62-1a59-4660-9515-aad1c170fb86" containerName="revoke-keys"
Nov 02 01:48:21.648888 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.648063    5639 cpu_manager.go:414] "containerMap: removing container" podUID="f02f9c71-728b-49af-9657-c3574c42f998" containerName="revoke-keys"
Nov 02 01:48:21.648888 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.648073    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f02f9c71-728b-49af-9657-c3574c42f998" containerName="revoke-keys"
Nov 02 01:48:21.648888 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.648345    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="8be73d62-1a59-4660-9515-aad1c170fb86" containerName="revoke-keys"
Nov 02 01:48:21.648888 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.648366    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="f02f9c71-728b-49af-9657-c3574c42f998" containerName="revoke-keys"
Nov 02 01:48:21.654230 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.654207    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e00cd-kz2jm"
Nov 02 01:48:21.678075 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.671328    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c966814-z97g5"]
Nov 02 01:48:21.688163 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.688133    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c966814-z97g5"
Nov 02 01:48:21.706681 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.706647    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:48:21.706868 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.706742    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:48:21.710154 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.710107    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:48:21.712312 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.712242    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:48:21.724040 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.724008    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f02408-f6kbp"]
Nov 02 01:48:21.734062 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.734031    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/092d32f7-43a4-4b4d-a890-9941f86515e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c966814-z97g5\" (UID: \"092d32f7-43a4-4b4d-a890-9941f86515e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c966814-z97g5"
Nov 02 01:48:21.735550 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.735518    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbp2c\" (UniqueName: \"kubernetes.io/projected/092d32f7-43a4-4b4d-a890-9941f86515e7-kube-api-access-sbp2c\") pod \"maas-api-revoke-sub-2c966814-z97g5\" (UID: \"092d32f7-43a4-4b4d-a890-9941f86515e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c966814-z97g5"
Nov 02 01:48:21.735820 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.735800    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7abc26b-f2f9-49fa-afd1-d42f64c0b458-maas-api-xks-ca\") pod \"maas-api-revoke-sub-372e00cd-kz2jm\" (UID: \"b7abc26b-f2f9-49fa-afd1-d42f64c0b458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e00cd-kz2jm"
Nov 02 01:48:21.735953 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.735937    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/092d32f7-43a4-4b4d-a890-9941f86515e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c966814-z97g5\" (UID: \"092d32f7-43a4-4b4d-a890-9941f86515e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c966814-z97g5"
Nov 02 01:48:21.736142 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.736124    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7abc26b-f2f9-49fa-afd1-d42f64c0b458-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-372e00cd-kz2jm\" (UID: \"b7abc26b-f2f9-49fa-afd1-d42f64c0b458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e00cd-kz2jm"
Nov 02 01:48:21.736250 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.736235    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptjq2\" (UniqueName: \"kubernetes.io/projected/b7abc26b-f2f9-49fa-afd1-d42f64c0b458-kube-api-access-ptjq2\") pod \"maas-api-revoke-sub-372e00cd-kz2jm\" (UID: \"b7abc26b-f2f9-49fa-afd1-d42f64c0b458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e00cd-kz2jm"
Nov 02 01:48:21.741437 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.741416    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-372e00cd-kz2jm"]
Nov 02 01:48:21.741699 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.741640    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f02408-f6kbp"
Nov 02 01:48:21.747720 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.747652    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c966814-z97g5"]
Nov 02 01:48:21.754374 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.754345    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f02408-f6kbp"]
Nov 02 01:48:21.837817 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.837784    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbp2c\" (UniqueName: \"kubernetes.io/projected/092d32f7-43a4-4b4d-a890-9941f86515e7-kube-api-access-sbp2c\") pod \"maas-api-revoke-sub-2c966814-z97g5\" (UID: \"092d32f7-43a4-4b4d-a890-9941f86515e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c966814-z97g5"
Nov 02 01:48:21.838166 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.838136    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7abc26b-f2f9-49fa-afd1-d42f64c0b458-maas-api-xks-ca\") pod \"maas-api-revoke-sub-372e00cd-kz2jm\" (UID: \"b7abc26b-f2f9-49fa-afd1-d42f64c0b458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e00cd-kz2jm"
Nov 02 01:48:21.838337 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.838318    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/092d32f7-43a4-4b4d-a890-9941f86515e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c966814-z97g5\" (UID: \"092d32f7-43a4-4b4d-a890-9941f86515e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c966814-z97g5"
Nov 02 01:48:21.838562 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.838541    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7abc26b-f2f9-49fa-afd1-d42f64c0b458-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-372e00cd-kz2jm\" (UID: \"b7abc26b-f2f9-49fa-afd1-d42f64c0b458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e00cd-kz2jm"
Nov 02 01:48:21.838691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.838658    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptjq2\" (UniqueName: \"kubernetes.io/projected/b7abc26b-f2f9-49fa-afd1-d42f64c0b458-kube-api-access-ptjq2\") pod \"maas-api-revoke-sub-372e00cd-kz2jm\" (UID: \"b7abc26b-f2f9-49fa-afd1-d42f64c0b458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e00cd-kz2jm"
Nov 02 01:48:21.838813 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.838797    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed7ff800-8da6-4dc6-9836-84affd85382a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65f02408-f6kbp\" (UID: \"ed7ff800-8da6-4dc6-9836-84affd85382a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f02408-f6kbp"
Nov 02 01:48:21.839002 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.838986    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/092d32f7-43a4-4b4d-a890-9941f86515e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c966814-z97g5\" (UID: \"092d32f7-43a4-4b4d-a890-9941f86515e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c966814-z97g5"
Nov 02 01:48:21.839144 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.839130    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmfcx\" (UniqueName: \"kubernetes.io/projected/ed7ff800-8da6-4dc6-9836-84affd85382a-kube-api-access-pmfcx\") pod \"maas-api-revoke-sub-65f02408-f6kbp\" (UID: \"ed7ff800-8da6-4dc6-9836-84affd85382a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f02408-f6kbp"
Nov 02 01:48:21.839350 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.839333    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed7ff800-8da6-4dc6-9836-84affd85382a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65f02408-f6kbp\" (UID: \"ed7ff800-8da6-4dc6-9836-84affd85382a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f02408-f6kbp"
Nov 02 01:48:21.841923 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.841885    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/092d32f7-43a4-4b4d-a890-9941f86515e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c966814-z97g5\" (UID: \"092d32f7-43a4-4b4d-a890-9941f86515e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c966814-z97g5"
Nov 02 01:48:21.844454 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.844405    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7abc26b-f2f9-49fa-afd1-d42f64c0b458-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-372e00cd-kz2jm\" (UID: \"b7abc26b-f2f9-49fa-afd1-d42f64c0b458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e00cd-kz2jm"
Nov 02 01:48:21.845978 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.845931    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/092d32f7-43a4-4b4d-a890-9941f86515e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c966814-z97g5\" (UID: \"092d32f7-43a4-4b4d-a890-9941f86515e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c966814-z97g5"
Nov 02 01:48:21.857455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.857428    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7abc26b-f2f9-49fa-afd1-d42f64c0b458-maas-api-xks-ca\") pod \"maas-api-revoke-sub-372e00cd-kz2jm\" (UID: \"b7abc26b-f2f9-49fa-afd1-d42f64c0b458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e00cd-kz2jm"
Nov 02 01:48:21.902705 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.902677    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbp2c\" (UniqueName: \"kubernetes.io/projected/092d32f7-43a4-4b4d-a890-9941f86515e7-kube-api-access-sbp2c\") pod \"maas-api-revoke-sub-2c966814-z97g5\" (UID: \"092d32f7-43a4-4b4d-a890-9941f86515e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c966814-z97g5"
Nov 02 01:48:21.909811 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.909748    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptjq2\" (UniqueName: \"kubernetes.io/projected/b7abc26b-f2f9-49fa-afd1-d42f64c0b458-kube-api-access-ptjq2\") pod \"maas-api-revoke-sub-372e00cd-kz2jm\" (UID: \"b7abc26b-f2f9-49fa-afd1-d42f64c0b458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e00cd-kz2jm"
Nov 02 01:48:21.943173 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.943134    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed7ff800-8da6-4dc6-9836-84affd85382a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65f02408-f6kbp\" (UID: \"ed7ff800-8da6-4dc6-9836-84affd85382a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f02408-f6kbp"
Nov 02 01:48:21.943420 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.943255    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmfcx\" (UniqueName: \"kubernetes.io/projected/ed7ff800-8da6-4dc6-9836-84affd85382a-kube-api-access-pmfcx\") pod \"maas-api-revoke-sub-65f02408-f6kbp\" (UID: \"ed7ff800-8da6-4dc6-9836-84affd85382a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f02408-f6kbp"
Nov 02 01:48:21.943420 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.943341    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed7ff800-8da6-4dc6-9836-84affd85382a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65f02408-f6kbp\" (UID: \"ed7ff800-8da6-4dc6-9836-84affd85382a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f02408-f6kbp"
Nov 02 01:48:21.944617 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.944589    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed7ff800-8da6-4dc6-9836-84affd85382a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65f02408-f6kbp\" (UID: \"ed7ff800-8da6-4dc6-9836-84affd85382a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f02408-f6kbp"
Nov 02 01:48:21.946223 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.946202    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed7ff800-8da6-4dc6-9836-84affd85382a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65f02408-f6kbp\" (UID: \"ed7ff800-8da6-4dc6-9836-84affd85382a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f02408-f6kbp"
Nov 02 01:48:21.981529 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.978010    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:21.981529 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.978579    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:21.984717 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.984678    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:48:21.991108 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:21.988875    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e00cd-kz2jm"
Nov 02 01:48:22.006181 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:22.006147    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmfcx\" (UniqueName: \"kubernetes.io/projected/ed7ff800-8da6-4dc6-9836-84affd85382a-kube-api-access-pmfcx\") pod \"maas-api-revoke-sub-65f02408-f6kbp\" (UID: \"ed7ff800-8da6-4dc6-9836-84affd85382a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f02408-f6kbp"
Nov 02 01:48:22.026301 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:22.026255    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c966814-z97g5"
Nov 02 01:48:22.072136 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:22.071537    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f02408-f6kbp"
Nov 02 01:48:22.397438 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:22.397407    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-372e00cd-kz2jm"]
Nov 02 01:48:22.398050 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:48:22.398019    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7abc26b_f2f9_49fa_afd1_d42f64c0b458.slice/crio-a82f5e977d8daaaeaeb5d4fd42f4596a5f38656fc110c64de80c68a2c24c7c37 WatchSource:0}: Error finding container a82f5e977d8daaaeaeb5d4fd42f4596a5f38656fc110c64de80c68a2c24c7c37: Status 404 returned error can't find the container with id a82f5e977d8daaaeaeb5d4fd42f4596a5f38656fc110c64de80c68a2c24c7c37
Nov 02 01:48:22.461403 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:48:22.461354    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod092d32f7_43a4_4b4d_a890_9941f86515e7.slice/crio-8b2b0b9c0b535dc7d84b8799bde8e19764b8bd94ee29662a2c693fceec8cc8a1 WatchSource:0}: Error finding container 8b2b0b9c0b535dc7d84b8799bde8e19764b8bd94ee29662a2c693fceec8cc8a1: Status 404 returned error can't find the container with id 8b2b0b9c0b535dc7d84b8799bde8e19764b8bd94ee29662a2c693fceec8cc8a1
Nov 02 01:48:22.462972 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:22.462941    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c966814-z97g5"]
Nov 02 01:48:22.495835 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:22.495806    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f02408-f6kbp"]
Nov 02 01:48:22.805072 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:22.805041    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e00cd-kz2jm" event={"ID":"b7abc26b-f2f9-49fa-afd1-d42f64c0b458","Type":"ContainerStarted","Data":"f7954afe2c6f0f03d1eaed6cd489ea08e0db4b89a55f448d18334a45b5aae657"}
Nov 02 01:48:22.805231 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:22.805105    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e00cd-kz2jm" event={"ID":"b7abc26b-f2f9-49fa-afd1-d42f64c0b458","Type":"ContainerStarted","Data":"a82f5e977d8daaaeaeb5d4fd42f4596a5f38656fc110c64de80c68a2c24c7c37"}
Nov 02 01:48:22.806722 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:22.806694    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c966814-z97g5" event={"ID":"092d32f7-43a4-4b4d-a890-9941f86515e7","Type":"ContainerStarted","Data":"8b2b0b9c0b535dc7d84b8799bde8e19764b8bd94ee29662a2c693fceec8cc8a1"}
Nov 02 01:48:22.808090 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:22.808067    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f02408-f6kbp" event={"ID":"ed7ff800-8da6-4dc6-9836-84affd85382a","Type":"ContainerStarted","Data":"ce2ee9021b37ed1af2ffe1a8b99d936c3c94b82add042ded0d30fa89ed188e81"}
Nov 02 01:48:22.921703 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:22.921645    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e00cd-kz2jm" podStartSLOduration=1.921627533 podStartE2EDuration="1.921627533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:22.919644648 +0000 UTC m=+1367.823326762" watchObservedRunningTime="2026-11-02 01:48:22.921627533 +0000 UTC m=+1367.825309648"
Nov 02 01:48:23.820856 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:23.820822    5639 generic.go:356] "Generic (PLEG): container finished" podID="092d32f7-43a4-4b4d-a890-9941f86515e7" containerID="90dbf4cfdebc679f3ebcb9c7e0d7b8691ced8d6c84d963242af85dc66753b5bc" exitCode=0
Nov 02 01:48:23.821295 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:23.820905    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c966814-z97g5" event={"ID":"092d32f7-43a4-4b4d-a890-9941f86515e7","Type":"ContainerDied","Data":"90dbf4cfdebc679f3ebcb9c7e0d7b8691ced8d6c84d963242af85dc66753b5bc"}
Nov 02 01:48:23.822748 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:23.822720    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f02408-f6kbp" event={"ID":"ed7ff800-8da6-4dc6-9836-84affd85382a","Type":"ContainerStarted","Data":"9e1c6b166e4853892eea71044805c738ad7461382af41f642c9cf73fcef4bafd"}
Nov 02 01:48:23.824449 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:23.824428    5639 generic.go:356] "Generic (PLEG): container finished" podID="b7abc26b-f2f9-49fa-afd1-d42f64c0b458" containerID="f7954afe2c6f0f03d1eaed6cd489ea08e0db4b89a55f448d18334a45b5aae657" exitCode=0
Nov 02 01:48:23.824524 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:23.824497    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e00cd-kz2jm" event={"ID":"b7abc26b-f2f9-49fa-afd1-d42f64c0b458","Type":"ContainerDied","Data":"f7954afe2c6f0f03d1eaed6cd489ea08e0db4b89a55f448d18334a45b5aae657"}
Nov 02 01:48:24.052229 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:24.052192    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68278e7c-8xtqp"]
Nov 02 01:48:24.056953 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:24.056927    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68278e7c-8xtqp"]
Nov 02 01:48:24.170568 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:24.170460    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f02408-f6kbp" podStartSLOduration=3.170444861 podStartE2EDuration="3.170444861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:24.166931798 +0000 UTC m=+1369.070613914" watchObservedRunningTime="2026-11-02 01:48:24.170444861 +0000 UTC m=+1369.074126982"
Nov 02 01:48:24.839293 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:24.839240    5639 generic.go:356] "Generic (PLEG): container finished" podID="ed7ff800-8da6-4dc6-9836-84affd85382a" containerID="9e1c6b166e4853892eea71044805c738ad7461382af41f642c9cf73fcef4bafd" exitCode=0
Nov 02 01:48:24.839776 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:24.839341    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f02408-f6kbp" event={"ID":"ed7ff800-8da6-4dc6-9836-84affd85382a","Type":"ContainerDied","Data":"9e1c6b166e4853892eea71044805c738ad7461382af41f642c9cf73fcef4bafd"}
Nov 02 01:48:25.030965 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.030931    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e00cd-kz2jm"
Nov 02 01:48:25.076352 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.076327    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c966814-z97g5"
Nov 02 01:48:25.112719 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.112635    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7abc26b-f2f9-49fa-afd1-d42f64c0b458-kube-api-access-ptjq2\" (UniqueName: \"kubernetes.io/projected/b7abc26b-f2f9-49fa-afd1-d42f64c0b458-kube-api-access-ptjq2\") pod \"b7abc26b-f2f9-49fa-afd1-d42f64c0b458\" (UID: \"b7abc26b-f2f9-49fa-afd1-d42f64c0b458\") "
Nov 02 01:48:25.112877 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.112750    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/092d32f7-43a4-4b4d-a890-9941f86515e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/092d32f7-43a4-4b4d-a890-9941f86515e7-maas-api-openshift-ca\") pod \"092d32f7-43a4-4b4d-a890-9941f86515e7\" (UID: \"092d32f7-43a4-4b4d-a890-9941f86515e7\") "
Nov 02 01:48:25.112962 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.112895    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/092d32f7-43a4-4b4d-a890-9941f86515e7-kube-api-access-sbp2c\" (UniqueName: \"kubernetes.io/projected/092d32f7-43a4-4b4d-a890-9941f86515e7-kube-api-access-sbp2c\") pod \"092d32f7-43a4-4b4d-a890-9941f86515e7\" (UID: \"092d32f7-43a4-4b4d-a890-9941f86515e7\") "
Nov 02 01:48:25.113021 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.112959    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7abc26b-f2f9-49fa-afd1-d42f64c0b458-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7abc26b-f2f9-49fa-afd1-d42f64c0b458-maas-api-xks-ca\") pod \"b7abc26b-f2f9-49fa-afd1-d42f64c0b458\" (UID: \"b7abc26b-f2f9-49fa-afd1-d42f64c0b458\") "
Nov 02 01:48:25.113118 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.113090    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/092d32f7-43a4-4b4d-a890-9941f86515e7-maas-api-openshift-ca" pod "092d32f7-43a4-4b4d-a890-9941f86515e7" (UID: "092d32f7-43a4-4b4d-a890-9941f86515e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:25.113197 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.113100    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7abc26b-f2f9-49fa-afd1-d42f64c0b458-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7abc26b-f2f9-49fa-afd1-d42f64c0b458-maas-api-openshift-ca\") pod \"b7abc26b-f2f9-49fa-afd1-d42f64c0b458\" (UID: \"b7abc26b-f2f9-49fa-afd1-d42f64c0b458\") "
Nov 02 01:48:25.113262 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.113197    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/092d32f7-43a4-4b4d-a890-9941f86515e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/092d32f7-43a4-4b4d-a890-9941f86515e7-maas-api-xks-ca\") pod \"092d32f7-43a4-4b4d-a890-9941f86515e7\" (UID: \"092d32f7-43a4-4b4d-a890-9941f86515e7\") "
Nov 02 01:48:25.113532 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.113443    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7abc26b-f2f9-49fa-afd1-d42f64c0b458-maas-api-openshift-ca" pod "b7abc26b-f2f9-49fa-afd1-d42f64c0b458" (UID: "b7abc26b-f2f9-49fa-afd1-d42f64c0b458"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:25.113969 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.113948    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/092d32f7-43a4-4b4d-a890-9941f86515e7-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:25.113969 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.113969    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7abc26b-f2f9-49fa-afd1-d42f64c0b458-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:25.115268 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.115245    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7abc26b-f2f9-49fa-afd1-d42f64c0b458-kube-api-access-ptjq2" pod "b7abc26b-f2f9-49fa-afd1-d42f64c0b458" (UID: "b7abc26b-f2f9-49fa-afd1-d42f64c0b458"). InnerVolumeSpecName "kube-api-access-ptjq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:25.115510 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.115484    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/092d32f7-43a4-4b4d-a890-9941f86515e7-maas-api-xks-ca" pod "092d32f7-43a4-4b4d-a890-9941f86515e7" (UID: "092d32f7-43a4-4b4d-a890-9941f86515e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:25.115737 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.115714    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7abc26b-f2f9-49fa-afd1-d42f64c0b458-maas-api-xks-ca" pod "b7abc26b-f2f9-49fa-afd1-d42f64c0b458" (UID: "b7abc26b-f2f9-49fa-afd1-d42f64c0b458"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:25.115737 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.115721    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092d32f7-43a4-4b4d-a890-9941f86515e7-kube-api-access-sbp2c" pod "092d32f7-43a4-4b4d-a890-9941f86515e7" (UID: "092d32f7-43a4-4b4d-a890-9941f86515e7"). InnerVolumeSpecName "kube-api-access-sbp2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:25.215094 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.215063    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbp2c\" (UniqueName: \"kubernetes.io/projected/092d32f7-43a4-4b4d-a890-9941f86515e7-kube-api-access-sbp2c\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:25.215094 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.215094    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7abc26b-f2f9-49fa-afd1-d42f64c0b458-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:25.215094 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.215106    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/092d32f7-43a4-4b4d-a890-9941f86515e7-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:25.215374 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.215115    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptjq2\" (UniqueName: \"kubernetes.io/projected/b7abc26b-f2f9-49fa-afd1-d42f64c0b458-kube-api-access-ptjq2\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:25.230259 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.230218    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c19baea-9ea4-4e66-99a1-056a3c93ad99" path="/var/lib/kubelet/pods/9c19baea-9ea4-4e66-99a1-056a3c93ad99/volumes"
Nov 02 01:48:25.850215 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.850182    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c966814-z97g5"
Nov 02 01:48:25.850215 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.850193    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c966814-z97g5" event={"ID":"092d32f7-43a4-4b4d-a890-9941f86515e7","Type":"ContainerDied","Data":"8b2b0b9c0b535dc7d84b8799bde8e19764b8bd94ee29662a2c693fceec8cc8a1"}
Nov 02 01:48:25.850757 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.850255    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b2b0b9c0b535dc7d84b8799bde8e19764b8bd94ee29662a2c693fceec8cc8a1"
Nov 02 01:48:25.852089 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.852066    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e00cd-kz2jm"
Nov 02 01:48:25.852232 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.852090    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e00cd-kz2jm" event={"ID":"b7abc26b-f2f9-49fa-afd1-d42f64c0b458","Type":"ContainerDied","Data":"a82f5e977d8daaaeaeb5d4fd42f4596a5f38656fc110c64de80c68a2c24c7c37"}
Nov 02 01:48:25.852232 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:25.852132    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a82f5e977d8daaaeaeb5d4fd42f4596a5f38656fc110c64de80c68a2c24c7c37"
Nov 02 01:48:26.055520 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:26.055485    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f02408-f6kbp"
Nov 02 01:48:26.155254 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:26.155165    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed7ff800-8da6-4dc6-9836-84affd85382a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed7ff800-8da6-4dc6-9836-84affd85382a-maas-api-xks-ca\") pod \"ed7ff800-8da6-4dc6-9836-84affd85382a\" (UID: \"ed7ff800-8da6-4dc6-9836-84affd85382a\") "
Nov 02 01:48:26.155254 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:26.155238    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed7ff800-8da6-4dc6-9836-84affd85382a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed7ff800-8da6-4dc6-9836-84affd85382a-maas-api-openshift-ca\") pod \"ed7ff800-8da6-4dc6-9836-84affd85382a\" (UID: \"ed7ff800-8da6-4dc6-9836-84affd85382a\") "
Nov 02 01:48:26.155508 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:26.155340    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed7ff800-8da6-4dc6-9836-84affd85382a-kube-api-access-pmfcx\" (UniqueName: \"kubernetes.io/projected/ed7ff800-8da6-4dc6-9836-84affd85382a-kube-api-access-pmfcx\") pod \"ed7ff800-8da6-4dc6-9836-84affd85382a\" (UID: \"ed7ff800-8da6-4dc6-9836-84affd85382a\") "
Nov 02 01:48:26.155676 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:26.155650    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed7ff800-8da6-4dc6-9836-84affd85382a-maas-api-openshift-ca" pod "ed7ff800-8da6-4dc6-9836-84affd85382a" (UID: "ed7ff800-8da6-4dc6-9836-84affd85382a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:26.155886 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:26.155869    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed7ff800-8da6-4dc6-9836-84affd85382a-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:26.157618 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:26.157594    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed7ff800-8da6-4dc6-9836-84affd85382a-maas-api-xks-ca" pod "ed7ff800-8da6-4dc6-9836-84affd85382a" (UID: "ed7ff800-8da6-4dc6-9836-84affd85382a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:26.157715 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:26.157593    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed7ff800-8da6-4dc6-9836-84affd85382a-kube-api-access-pmfcx" pod "ed7ff800-8da6-4dc6-9836-84affd85382a" (UID: "ed7ff800-8da6-4dc6-9836-84affd85382a"). InnerVolumeSpecName "kube-api-access-pmfcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:26.257181 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:26.257149    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed7ff800-8da6-4dc6-9836-84affd85382a-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:26.257181 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:26.257176    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmfcx\" (UniqueName: \"kubernetes.io/projected/ed7ff800-8da6-4dc6-9836-84affd85382a-kube-api-access-pmfcx\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:26.863303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:26.863123    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f02408-f6kbp" event={"ID":"ed7ff800-8da6-4dc6-9836-84affd85382a","Type":"ContainerDied","Data":"ce2ee9021b37ed1af2ffe1a8b99d936c3c94b82add042ded0d30fa89ed188e81"}
Nov 02 01:48:26.863303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:26.863183    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce2ee9021b37ed1af2ffe1a8b99d936c3c94b82add042ded0d30fa89ed188e81"
Nov 02 01:48:26.863303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:26.863136    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f02408-f6kbp"
Nov 02 01:48:27.503117 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:48:27.503077    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:48:27.777371 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:27.776657    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7048-4e00a9"]
Nov 02 01:48:27.781308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:27.777750    5639 cpu_manager.go:414] "containerMap: removing container" podUID="ed7ff800-8da6-4dc6-9836-84affd85382a" containerName="revoke-keys"
Nov 02 01:48:27.781308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:27.777777    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed7ff800-8da6-4dc6-9836-84affd85382a" containerName="revoke-keys"
Nov 02 01:48:27.781308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:27.777804    5639 cpu_manager.go:414] "containerMap: removing container" podUID="b7abc26b-f2f9-49fa-afd1-d42f64c0b458" containerName="revoke-keys"
Nov 02 01:48:27.781308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:27.777811    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7abc26b-f2f9-49fa-afd1-d42f64c0b458" containerName="revoke-keys"
Nov 02 01:48:27.781308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:27.777828    5639 cpu_manager.go:414] "containerMap: removing container" podUID="092d32f7-43a4-4b4d-a890-9941f86515e7" containerName="revoke-keys"
Nov 02 01:48:27.781308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:27.777835    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="092d32f7-43a4-4b4d-a890-9941f86515e7" containerName="revoke-keys"
Nov 02 01:48:27.781308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:27.778015    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed7ff800-8da6-4dc6-9836-84affd85382a" containerName="revoke-keys"
Nov 02 01:48:27.781308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:27.778031    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7abc26b-f2f9-49fa-afd1-d42f64c0b458" containerName="revoke-keys"
Nov 02 01:48:27.781308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:27.778044    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="092d32f7-43a4-4b4d-a890-9941f86515e7" containerName="revoke-keys"
Nov 02 01:48:27.785304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:27.782709    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7048-4e00a9"
Nov 02 01:48:27.819346 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:27.802612    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7048-4e00a9"]
Nov 02 01:48:27.861307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:27.847323    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-n9czn\""
Nov 02 01:48:28.056370 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:27.986570    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt924\" (UniqueName: \"kubernetes.io/projected/8ed19135-0549-455d-aacd-fbdbc1cc8858-kube-api-access-vt924\") pod \"test-curl-7048-4e00a9\" (UID: \"8ed19135-0549-455d-aacd-fbdbc1cc8858\") " pod="opendatahub/test-curl-7048-4e00a9"
Nov 02 01:48:28.281307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:28.260897    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt924\" (UniqueName: \"kubernetes.io/projected/8ed19135-0549-455d-aacd-fbdbc1cc8858-kube-api-access-vt924\") pod \"test-curl-7048-4e00a9\" (UID: \"8ed19135-0549-455d-aacd-fbdbc1cc8858\") " pod="opendatahub/test-curl-7048-4e00a9"
Nov 02 01:48:28.435005 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:28.434971    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt924\" (UniqueName: \"kubernetes.io/projected/8ed19135-0549-455d-aacd-fbdbc1cc8858-kube-api-access-vt924\") pod \"test-curl-7048-4e00a9\" (UID: \"8ed19135-0549-455d-aacd-fbdbc1cc8858\") " pod="opendatahub/test-curl-7048-4e00a9"
Nov 02 01:48:28.554313 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:28.538648    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7048-4e00a9"
Nov 02 01:48:28.627308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:28.621429    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:28.627308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:28.622838    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:48:28.655304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:28.643375    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:28.730314 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:28.725396    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:28.902715 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:28.900587    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:28.952466 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:28.952361    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Nov 02 01:48:29.007438 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:29.007392    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:29.020421 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:29.020369    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/health\": context deadline exceeded"
Nov 02 01:48:29.053386 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:29.053342    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:29.186872 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:29.186823    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:29.631556 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:29.631452    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:29.647347 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:29.635533    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:29.647347 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:29.636245    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 02 01:48:29.647347 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:29.636633    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:48:29.647347 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:29.636690    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:48:29.663941 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:29.661621    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:29.663941 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:29.662012    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:48:29.737315 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:29.733408    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:48:29.737315 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:29.733945    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:48:29.939628 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:29.908552    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:29.939628 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:29.912591    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:48:30.103260 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:30.100328    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded"
Nov 02 01:48:30.103260 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:30.102722    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": context deadline exceeded"
Nov 02 01:48:30.103600 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:30.103576    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:30.105719 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:30.105687    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:48:30.107242 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:30.107220    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:48:30.107471 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:30.107455    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:48:30.113092 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:30.113067    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:48:30.220733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:30.220637    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 02 01:48:30.222556 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:30.222531    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:48:30.471175 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:30.448376    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:48:30.649804 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:30.649757    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:30.673635 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:30.670560    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:30.673635 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:30.672357    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:30.688181 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:30.688149    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8408d89d-f6p4s"]
Nov 02 01:48:30.732686 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:30.731248    5639 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:48:30.787371 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:30.772159    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:48:30.968759 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:30.968718    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:31.127309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:31.112168    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": context deadline exceeded"
Nov 02 01:48:31.127309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:31.112536    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Nov 02 01:48:31.127309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:31.112797    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:48:31.161303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:31.136222    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8408d89d-f6p4s"]
Nov 02 01:48:31.263093 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:31.263045    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 02 01:48:31.453504 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:31.436190    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:31.489030 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:31.455062    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:48:31.514539 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:31.514491    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5" podUID="50ac256a-961a-4ff2-b454-06a201660154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:31.708365 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:31.685553    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:31.708365 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:31.685967    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:31.832474 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:31.832429    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:31.882313 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:31.869056    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6578a06-3971-4811-9505-ee4eaf1a5627" path="/var/lib/kubelet/pods/c6578a06-3971-4811-9505-ee4eaf1a5627/volumes"
Nov 02 01:48:31.925306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:31.888265    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7048-4e00a9"]
Nov 02 01:48:31.997363 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:31.984696    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:32.118315 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:32.095453    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7048-4e00a9" event={"ID":"8ed19135-0549-455d-aacd-fbdbc1cc8858","Type":"ContainerStarted","Data":"b75523052f6ca89f679bf02822280d5f568d984e13fa2231ef3f6b9b647b643c"}
Nov 02 01:48:32.119901 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:32.119858    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:32.121530 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:32.121489    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:32.477928 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:32.136301    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Nov 02 01:48:32.503803 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:32.503756    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:48:32.903318 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:32.512259    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:48:32.907306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:32.578924    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 02 01:48:33.054381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:32.771741    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:33.082649 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:32.771786    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:48:33.083227 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:32.908888    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:33.087815 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:33.004235    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:48:33.116857 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:33.116632    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:33.280925 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:33.280707    5639 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:48:33.281322 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:33.281260    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp" podUID="02d298db-5bc9-469a-af6b-ba7fa45f8602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:33.512697 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:33.477711    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:33.674335 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:33.529845    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:48:33.679729 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:33.553325    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:33.680652 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:33.572978    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:33.687666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:33.672920    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:48:33.687869 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:33.687839    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h" podUID="1640ee78-dc59-4afb-a7f6-e130f60cdf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:33.688020 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:33.672964    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:48:33.688145 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:33.688112    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp" podUID="945be3db-df2f-4ef9-9609-a7890bee84ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:34.035705 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.035594    5639 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:48:34.039304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.036020    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f" podUID="c4443617-f4c0-40c4-a3bf-0b6241d6ce97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:34.077634 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.077588    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:34.098052 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.098008    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:34.099157 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.099122    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 02 01:48:34.099954 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.099926    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:48:34.100567 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.100544    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:48:34.156683 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.153422    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:34.304894 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.304775    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:48:34.307238 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.307201    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:48:34.421622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.421586    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-278149b1-fk6vc"]
Nov 02 01:48:34.421931 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.421914    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:48:34.422111 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.422089    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:48:34.478303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.474601    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-278149b1-fk6vc"]
Nov 02 01:48:34.700165 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.700080    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:48:34.707688 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.705005    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:34.719626 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.708684    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:48:34.719626 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.714023    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:48:34.731377 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.726502    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:48:34.745244 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.743092    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:48:34.813632 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.813597    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ecd9882-r6t4x"]
Nov 02 01:48:34.818100 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.818077    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecd9882-r6t4x"
Nov 02 01:48:34.873458 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.873426    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ecd9882-r6t4x"]
Nov 02 01:48:34.874179 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.874151    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:48:34.875765 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.875743    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:48:34.875765 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.875747    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:48:34.875912 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.875786    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:48:34.921379 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.921345    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc2e0035-2bb2-47cf-a07a-78c060dc9ee3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ecd9882-r6t4x\" (UID: \"dc2e0035-2bb2-47cf-a07a-78c060dc9ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecd9882-r6t4x"
Nov 02 01:48:34.921681 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.921655    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b99r\" (UniqueName: \"kubernetes.io/projected/dc2e0035-2bb2-47cf-a07a-78c060dc9ee3-kube-api-access-9b99r\") pod \"maas-api-revoke-sub-1ecd9882-r6t4x\" (UID: \"dc2e0035-2bb2-47cf-a07a-78c060dc9ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecd9882-r6t4x"
Nov 02 01:48:34.921836 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.921814    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc2e0035-2bb2-47cf-a07a-78c060dc9ee3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ecd9882-r6t4x\" (UID: \"dc2e0035-2bb2-47cf-a07a-78c060dc9ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecd9882-r6t4x"
Nov 02 01:48:34.944992 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.944944    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b228babd-49ckm"]
Nov 02 01:48:34.949720 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:34.949695    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b228babd-49ckm"
Nov 02 01:48:35.004682 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.004629    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b228babd-49ckm"]
Nov 02 01:48:35.022936 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.022905    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b99r\" (UniqueName: \"kubernetes.io/projected/dc2e0035-2bb2-47cf-a07a-78c060dc9ee3-kube-api-access-9b99r\") pod \"maas-api-revoke-sub-1ecd9882-r6t4x\" (UID: \"dc2e0035-2bb2-47cf-a07a-78c060dc9ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecd9882-r6t4x"
Nov 02 01:48:35.023114 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.022998    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc2e0035-2bb2-47cf-a07a-78c060dc9ee3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ecd9882-r6t4x\" (UID: \"dc2e0035-2bb2-47cf-a07a-78c060dc9ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecd9882-r6t4x"
Nov 02 01:48:35.023114 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.023069    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc2e0035-2bb2-47cf-a07a-78c060dc9ee3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ecd9882-r6t4x\" (UID: \"dc2e0035-2bb2-47cf-a07a-78c060dc9ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecd9882-r6t4x"
Nov 02 01:48:35.023991 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.023966    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc2e0035-2bb2-47cf-a07a-78c060dc9ee3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ecd9882-r6t4x\" (UID: \"dc2e0035-2bb2-47cf-a07a-78c060dc9ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecd9882-r6t4x"
Nov 02 01:48:35.025857 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.025831    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc2e0035-2bb2-47cf-a07a-78c060dc9ee3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ecd9882-r6t4x\" (UID: \"dc2e0035-2bb2-47cf-a07a-78c060dc9ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecd9882-r6t4x"
Nov 02 01:48:35.085178 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.085140    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b99r\" (UniqueName: \"kubernetes.io/projected/dc2e0035-2bb2-47cf-a07a-78c060dc9ee3-kube-api-access-9b99r\") pod \"maas-api-revoke-sub-1ecd9882-r6t4x\" (UID: \"dc2e0035-2bb2-47cf-a07a-78c060dc9ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecd9882-r6t4x"
Nov 02 01:48:35.125055 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.125017    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30d6f81a-329d-4cb7-b131-27a605f0ad07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b228babd-49ckm\" (UID: \"30d6f81a-329d-4cb7-b131-27a605f0ad07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b228babd-49ckm"
Nov 02 01:48:35.125227 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.125208    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30d6f81a-329d-4cb7-b131-27a605f0ad07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b228babd-49ckm\" (UID: \"30d6f81a-329d-4cb7-b131-27a605f0ad07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b228babd-49ckm"
Nov 02 01:48:35.125344 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.125327    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9927\" (UniqueName: \"kubernetes.io/projected/30d6f81a-329d-4cb7-b131-27a605f0ad07-kube-api-access-m9927\") pod \"maas-api-revoke-sub-b228babd-49ckm\" (UID: \"30d6f81a-329d-4cb7-b131-27a605f0ad07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b228babd-49ckm"
Nov 02 01:48:35.142190 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.142134    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecd9882-r6t4x"
Nov 02 01:48:35.226683 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.226648    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30d6f81a-329d-4cb7-b131-27a605f0ad07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b228babd-49ckm\" (UID: \"30d6f81a-329d-4cb7-b131-27a605f0ad07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b228babd-49ckm"
Nov 02 01:48:35.226893 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.226798    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9927\" (UniqueName: \"kubernetes.io/projected/30d6f81a-329d-4cb7-b131-27a605f0ad07-kube-api-access-m9927\") pod \"maas-api-revoke-sub-b228babd-49ckm\" (UID: \"30d6f81a-329d-4cb7-b131-27a605f0ad07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b228babd-49ckm"
Nov 02 01:48:35.227007 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.226966    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30d6f81a-329d-4cb7-b131-27a605f0ad07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b228babd-49ckm\" (UID: \"30d6f81a-329d-4cb7-b131-27a605f0ad07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b228babd-49ckm"
Nov 02 01:48:35.229003 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.228222    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30d6f81a-329d-4cb7-b131-27a605f0ad07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b228babd-49ckm\" (UID: \"30d6f81a-329d-4cb7-b131-27a605f0ad07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b228babd-49ckm"
Nov 02 01:48:35.230159 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.230075    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fd58675-c64e-4a5f-bb11-a87dd9f49852" path="/var/lib/kubelet/pods/8fd58675-c64e-4a5f-bb11-a87dd9f49852/volumes"
Nov 02 01:48:35.232347 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.232324    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30d6f81a-329d-4cb7-b131-27a605f0ad07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b228babd-49ckm\" (UID: \"30d6f81a-329d-4cb7-b131-27a605f0ad07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b228babd-49ckm"
Nov 02 01:48:35.295214 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.294920    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9927\" (UniqueName: \"kubernetes.io/projected/30d6f81a-329d-4cb7-b131-27a605f0ad07-kube-api-access-m9927\") pod \"maas-api-revoke-sub-b228babd-49ckm\" (UID: \"30d6f81a-329d-4cb7-b131-27a605f0ad07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b228babd-49ckm"
Nov 02 01:48:35.496984 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.496899    5639 scope.go:122] "RemoveContainer" containerID="bc93aa7671586f9fbe46585735f2c72c814cf28a6d9d75a77311ab20c299ee07"
Nov 02 01:48:35.511401 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.511378    5639 scope.go:122] "RemoveContainer" containerID="a4cbbc43e75ef28929bde42f2983e50b1ab0f1c6e6aa6545a0abc665293e85e1"
Nov 02 01:48:35.527298 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.526893    5639 scope.go:122] "RemoveContainer" containerID="f80a4dfd341b8781c7b51b2f5d5e686f58b71ffe4bc07b4b2609f25ad7e29ea7"
Nov 02 01:48:35.540991 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.540963    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ecd9882-r6t4x"]
Nov 02 01:48:35.546820 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.546450    5639 scope.go:122] "RemoveContainer" containerID="1a7ed6f01703f7b64b0531a41289c6f2ee045bfb8f58ec0bc8f3ef8a7827b6d8"
Nov 02 01:48:35.563585 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.563560    5639 scope.go:122] "RemoveContainer" containerID="1afba297197ec82658a3c3b5438e1a606423301874665d95e6d1bed50927d0e6"
Nov 02 01:48:35.579629 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.579605    5639 scope.go:122] "RemoveContainer" containerID="1a18c92b2d5e09cfb06ce5c250e98c5287fdc0de5c75088fa3224be92305bedc"
Nov 02 01:48:35.592809 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.592781    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b228babd-49ckm"
Nov 02 01:48:35.895554 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:35.895520    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecd9882-r6t4x" event={"ID":"dc2e0035-2bb2-47cf-a07a-78c060dc9ee3","Type":"ContainerStarted","Data":"2f2384e999045f99f3f65d0e0c1189a498b29ddb0cff355dd5e1d6ae144d6062"}
Nov 02 01:48:36.014666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:36.014632    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b228babd-49ckm"]
Nov 02 01:48:36.018496 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:48:36.018465    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d6f81a_329d_4cb7_b131_27a605f0ad07.slice/crio-613eaf17648493e8e200a6fcf208ae97a4b914ddedf7d77d7b2d6909d1de9bf1 WatchSource:0}: Error finding container 613eaf17648493e8e200a6fcf208ae97a4b914ddedf7d77d7b2d6909d1de9bf1: Status 404 returned error can't find the container with id 613eaf17648493e8e200a6fcf208ae97a4b914ddedf7d77d7b2d6909d1de9bf1
Nov 02 01:48:36.908560 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:36.908492    5639 generic.go:356] "Generic (PLEG): container finished" podID="dc2e0035-2bb2-47cf-a07a-78c060dc9ee3" containerID="141964ec7fb312d76de05c700dca19346c8da659ca318fe11398da1d30b9b206" exitCode=0
Nov 02 01:48:36.909050 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:36.908576    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecd9882-r6t4x" event={"ID":"dc2e0035-2bb2-47cf-a07a-78c060dc9ee3","Type":"ContainerDied","Data":"141964ec7fb312d76de05c700dca19346c8da659ca318fe11398da1d30b9b206"}
Nov 02 01:48:36.911815 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:36.911545    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b228babd-49ckm" event={"ID":"30d6f81a-329d-4cb7-b131-27a605f0ad07","Type":"ContainerStarted","Data":"8049a0bd26e1297fcf415172077af6a4ccf093155e08bf64d87dcf7b2b46ab87"}
Nov 02 01:48:36.911815 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:36.911602    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b228babd-49ckm" event={"ID":"30d6f81a-329d-4cb7-b131-27a605f0ad07","Type":"ContainerStarted","Data":"613eaf17648493e8e200a6fcf208ae97a4b914ddedf7d77d7b2d6909d1de9bf1"}
Nov 02 01:48:37.138571 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:37.138363    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b228babd-49ckm" podStartSLOduration=3.138344178 podStartE2EDuration="3.138344178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:37.13796094 +0000 UTC m=+1382.041643086" watchObservedRunningTime="2026-11-02 01:48:37.138344178 +0000 UTC m=+1382.042026293"
Nov 02 01:48:37.922637 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:37.922596    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7048-4e00a9" event={"ID":"8ed19135-0549-455d-aacd-fbdbc1cc8858","Type":"ContainerStarted","Data":"d36428fc5412f1a9d000cb958c50c5a06514ef15503a5dc384a3070c91b06671"}
Nov 02 01:48:37.924424 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:37.924400    5639 generic.go:356] "Generic (PLEG): container finished" podID="30d6f81a-329d-4cb7-b131-27a605f0ad07" containerID="8049a0bd26e1297fcf415172077af6a4ccf093155e08bf64d87dcf7b2b46ab87" exitCode=0
Nov 02 01:48:37.924553 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:37.924487    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b228babd-49ckm" event={"ID":"30d6f81a-329d-4cb7-b131-27a605f0ad07","Type":"ContainerDied","Data":"8049a0bd26e1297fcf415172077af6a4ccf093155e08bf64d87dcf7b2b46ab87"}
Nov 02 01:48:38.040558 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:38.040429    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7048-4e00a9" podStartSLOduration=4.730301341 podStartE2EDuration="11.040410269s" totalImagesPullingTime="6.310108928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:38.035956945 +0000 UTC m=+1382.939639059" watchObservedRunningTime="2026-11-02 01:48:38.040410269 +0000 UTC m=+1382.944092384"
Nov 02 01:48:38.085144 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:38.085117    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecd9882-r6t4x"
Nov 02 01:48:38.110181 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:38.110148    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc2e0035-2bb2-47cf-a07a-78c060dc9ee3-kube-api-access-9b99r\" (UniqueName: \"kubernetes.io/projected/dc2e0035-2bb2-47cf-a07a-78c060dc9ee3-kube-api-access-9b99r\") pod \"dc2e0035-2bb2-47cf-a07a-78c060dc9ee3\" (UID: \"dc2e0035-2bb2-47cf-a07a-78c060dc9ee3\") "
Nov 02 01:48:38.110402 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:38.110230    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc2e0035-2bb2-47cf-a07a-78c060dc9ee3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc2e0035-2bb2-47cf-a07a-78c060dc9ee3-maas-api-openshift-ca\") pod \"dc2e0035-2bb2-47cf-a07a-78c060dc9ee3\" (UID: \"dc2e0035-2bb2-47cf-a07a-78c060dc9ee3\") "
Nov 02 01:48:38.110402 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:38.110361    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc2e0035-2bb2-47cf-a07a-78c060dc9ee3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc2e0035-2bb2-47cf-a07a-78c060dc9ee3-maas-api-xks-ca\") pod \"dc2e0035-2bb2-47cf-a07a-78c060dc9ee3\" (UID: \"dc2e0035-2bb2-47cf-a07a-78c060dc9ee3\") "
Nov 02 01:48:38.110635 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:38.110611    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc2e0035-2bb2-47cf-a07a-78c060dc9ee3-maas-api-openshift-ca" pod "dc2e0035-2bb2-47cf-a07a-78c060dc9ee3" (UID: "dc2e0035-2bb2-47cf-a07a-78c060dc9ee3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:38.110880 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:38.110865    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc2e0035-2bb2-47cf-a07a-78c060dc9ee3-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:38.112524 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:38.112497    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc2e0035-2bb2-47cf-a07a-78c060dc9ee3-maas-api-xks-ca" pod "dc2e0035-2bb2-47cf-a07a-78c060dc9ee3" (UID: "dc2e0035-2bb2-47cf-a07a-78c060dc9ee3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:38.112621 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:38.112530    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc2e0035-2bb2-47cf-a07a-78c060dc9ee3-kube-api-access-9b99r" pod "dc2e0035-2bb2-47cf-a07a-78c060dc9ee3" (UID: "dc2e0035-2bb2-47cf-a07a-78c060dc9ee3"). InnerVolumeSpecName "kube-api-access-9b99r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:38.212550 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:38.212473    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b99r\" (UniqueName: \"kubernetes.io/projected/dc2e0035-2bb2-47cf-a07a-78c060dc9ee3-kube-api-access-9b99r\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:38.212550 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:38.212497    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc2e0035-2bb2-47cf-a07a-78c060dc9ee3-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:38.934351 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:38.934323    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecd9882-r6t4x"
Nov 02 01:48:38.934775 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:38.934321    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecd9882-r6t4x" event={"ID":"dc2e0035-2bb2-47cf-a07a-78c060dc9ee3","Type":"ContainerDied","Data":"2f2384e999045f99f3f65d0e0c1189a498b29ddb0cff355dd5e1d6ae144d6062"}
Nov 02 01:48:38.934775 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:38.934462    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f2384e999045f99f3f65d0e0c1189a498b29ddb0cff355dd5e1d6ae144d6062"
Nov 02 01:48:39.113080 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.113047    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b228babd-49ckm"
Nov 02 01:48:39.227149 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.227071    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30d6f81a-329d-4cb7-b131-27a605f0ad07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30d6f81a-329d-4cb7-b131-27a605f0ad07-maas-api-xks-ca\") pod \"30d6f81a-329d-4cb7-b131-27a605f0ad07\" (UID: \"30d6f81a-329d-4cb7-b131-27a605f0ad07\") "
Nov 02 01:48:39.227328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.227175    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30d6f81a-329d-4cb7-b131-27a605f0ad07-kube-api-access-m9927\" (UniqueName: \"kubernetes.io/projected/30d6f81a-329d-4cb7-b131-27a605f0ad07-kube-api-access-m9927\") pod \"30d6f81a-329d-4cb7-b131-27a605f0ad07\" (UID: \"30d6f81a-329d-4cb7-b131-27a605f0ad07\") "
Nov 02 01:48:39.227424 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.227400    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d6f81a-329d-4cb7-b131-27a605f0ad07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30d6f81a-329d-4cb7-b131-27a605f0ad07-maas-api-openshift-ca\") pod \"30d6f81a-329d-4cb7-b131-27a605f0ad07\" (UID: \"30d6f81a-329d-4cb7-b131-27a605f0ad07\") "
Nov 02 01:48:39.227891 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.227815    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d6f81a-329d-4cb7-b131-27a605f0ad07-maas-api-openshift-ca" pod "30d6f81a-329d-4cb7-b131-27a605f0ad07" (UID: "30d6f81a-329d-4cb7-b131-27a605f0ad07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:39.228929 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.228200    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30d6f81a-329d-4cb7-b131-27a605f0ad07-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:39.230842 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.230777    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d6f81a-329d-4cb7-b131-27a605f0ad07-kube-api-access-m9927" pod "30d6f81a-329d-4cb7-b131-27a605f0ad07" (UID: "30d6f81a-329d-4cb7-b131-27a605f0ad07"). InnerVolumeSpecName "kube-api-access-m9927". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:39.231489 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.231464    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30d6f81a-329d-4cb7-b131-27a605f0ad07-maas-api-xks-ca" pod "30d6f81a-329d-4cb7-b131-27a605f0ad07" (UID: "30d6f81a-329d-4cb7-b131-27a605f0ad07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:39.329807 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.329774    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9927\" (UniqueName: \"kubernetes.io/projected/30d6f81a-329d-4cb7-b131-27a605f0ad07-kube-api-access-m9927\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:39.329807 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.329806    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30d6f81a-329d-4cb7-b131-27a605f0ad07-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:39.498460 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.498373    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7048-4e00a9"]
Nov 02 01:48:39.504191 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.504156    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7048-4e00a9"]
Nov 02 01:48:39.661793 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.661748    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"]
Nov 02 01:48:39.663408 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.663381    5639 cpu_manager.go:414] "containerMap: removing container" podUID="dc2e0035-2bb2-47cf-a07a-78c060dc9ee3" containerName="revoke-keys"
Nov 02 01:48:39.663408 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.663411    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc2e0035-2bb2-47cf-a07a-78c060dc9ee3" containerName="revoke-keys"
Nov 02 01:48:39.663601 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.663427    5639 cpu_manager.go:414] "containerMap: removing container" podUID="30d6f81a-329d-4cb7-b131-27a605f0ad07" containerName="revoke-keys"
Nov 02 01:48:39.663601 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.663437    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d6f81a-329d-4cb7-b131-27a605f0ad07" containerName="revoke-keys"
Nov 02 01:48:39.663601 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.663501    5639 cpu_manager.go:414] "containerMap: removing container" podUID="8ed19135-0549-455d-aacd-fbdbc1cc8858" containerName="test-curl-7048-4e00a9"
Nov 02 01:48:39.663601 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.663508    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ed19135-0549-455d-aacd-fbdbc1cc8858" containerName="test-curl-7048-4e00a9"
Nov 02 01:48:39.663867 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.663853    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ed19135-0549-455d-aacd-fbdbc1cc8858" containerName="test-curl-7048-4e00a9"
Nov 02 01:48:39.663914 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.663895    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc2e0035-2bb2-47cf-a07a-78c060dc9ee3" containerName="revoke-keys"
Nov 02 01:48:39.664040 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.663915    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d6f81a-329d-4cb7-b131-27a605f0ad07" containerName="revoke-keys"
Nov 02 01:48:39.669182 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.669160    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.733035 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.732993    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"]
Nov 02 01:48:39.743406 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.743369    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.743583 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.743458    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxdp5\" (UniqueName: \"kubernetes.io/projected/3da5e6df-b15c-4209-9cec-624599f6713d-kube-api-access-nxdp5\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.743627 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.743594    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.743743 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.743724    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3da5e6df-b15c-4209-9cec-624599f6713d-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.743777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.743764    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.743957 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.743935    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.744006 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.743991    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.744205 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.744180    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3da5e6df-b15c-4209-9cec-624599f6713d-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.744338 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.744318    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3da5e6df-b15c-4209-9cec-624599f6713d-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.845428 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.845366    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.845629 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.845438    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.845629 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.845525    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3da5e6df-b15c-4209-9cec-624599f6713d-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.845629 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.845569    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3da5e6df-b15c-4209-9cec-624599f6713d-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.845629 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.845604    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.845824 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.845649    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxdp5\" (UniqueName: \"kubernetes.io/projected/3da5e6df-b15c-4209-9cec-624599f6713d-kube-api-access-nxdp5\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.845824 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.845694    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.845824 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.845751    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3da5e6df-b15c-4209-9cec-624599f6713d-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.845824 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.845777    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.846033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.845891    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.846138 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.846115    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.846201 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.846167    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.846263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.846209    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.846708 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.846686    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3da5e6df-b15c-4209-9cec-624599f6713d-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.848918 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.848888    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3da5e6df-b15c-4209-9cec-624599f6713d-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.849146 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.849126    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.909368 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.909333    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3da5e6df-b15c-4209-9cec-624599f6713d-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.909524 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.909433    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxdp5\" (UniqueName: \"kubernetes.io/projected/3da5e6df-b15c-4209-9cec-624599f6713d-kube-api-access-nxdp5\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.982050 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.982014    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"]
Nov 02 01:48:39.991909 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.991492    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:39.994176 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.993925    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:39.998316 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.994538    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7048-4e00a9" podUID="8ed19135-0549-455d-aacd-fbdbc1cc8858" containerName="test-curl-7048-4e00a9" containerID="cri-o://d36428fc5412f1a9d000cb958c50c5a06514ef15503a5dc384a3070c91b06671" gracePeriod=2
Nov 02 01:48:39.998316 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.994927    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b228babd-49ckm"
Nov 02 01:48:39.998316 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.997244    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b228babd-49ckm" event={"ID":"30d6f81a-329d-4cb7-b131-27a605f0ad07","Type":"ContainerDied","Data":"613eaf17648493e8e200a6fcf208ae97a4b914ddedf7d77d7b2d6909d1de9bf1"}
Nov 02 01:48:39.998316 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:39.997315    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="613eaf17648493e8e200a6fcf208ae97a4b914ddedf7d77d7b2d6909d1de9bf1"
Nov 02 01:48:40.070878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.066651    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"]
Nov 02 01:48:40.070878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.069841    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47285028-a487-437d-8cd9-63221ffca4d6-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.070878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.069902    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.070878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.069980    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.070878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.070116    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.070878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.070178    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.070878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.070199    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47285028-a487-437d-8cd9-63221ffca4d6-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.070878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.070305    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.070878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.070408    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47285028-a487-437d-8cd9-63221ffca4d6-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.070878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.070433    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7k7s\" (UniqueName: \"kubernetes.io/projected/47285028-a487-437d-8cd9-63221ffca4d6-kube-api-access-g7k7s\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.167266 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:48:40.167230    5639 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7048-4e00a9\" is forbidden: User \"system:node:ip-10-0-136-53.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-53.us-west-1.compute.internal' and this object" podUID="8ed19135-0549-455d-aacd-fbdbc1cc8858" pod="opendatahub/test-curl-7048-4e00a9"
Nov 02 01:48:40.174038 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.172223    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.174038 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.172326    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47285028-a487-437d-8cd9-63221ffca4d6-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.174038 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.172412    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.174038 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.172503    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47285028-a487-437d-8cd9-63221ffca4d6-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.174038 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.172534    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7k7s\" (UniqueName: \"kubernetes.io/projected/47285028-a487-437d-8cd9-63221ffca4d6-kube-api-access-g7k7s\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.174038 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.172603    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47285028-a487-437d-8cd9-63221ffca4d6-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.174038 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.172646    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.174038 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.172724    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.174038 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.172879    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.174038 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.173365    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.174038 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.173610    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.183685 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.183121    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.183874 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.183704    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.183874 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.183812    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47285028-a487-437d-8cd9-63221ffca4d6-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.184968 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.184942    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47285028-a487-437d-8cd9-63221ffca4d6-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.186799 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.186763    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.218370 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:48:40.218331    5639 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7048-4e00a9\" is forbidden: User \"system:node:ip-10-0-136-53.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-53.us-west-1.compute.internal' and this object" podUID="8ed19135-0549-455d-aacd-fbdbc1cc8858" pod="opendatahub/test-curl-7048-4e00a9"
Nov 02 01:48:40.246454 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.246423    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47285028-a487-437d-8cd9-63221ffca4d6-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.246616 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.246506    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7k7s\" (UniqueName: \"kubernetes.io/projected/47285028-a487-437d-8cd9-63221ffca4d6-kube-api-access-g7k7s\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.467582 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.467493    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:40.581337 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.575325    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"]
Nov 02 01:48:40.588658 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.588618    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:48:40.588787 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.588705    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:48:40.588787 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.588748    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:48:40.855541 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.855509    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"]
Nov 02 01:48:40.858891 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.858855    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:48:40.858995 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.858939    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:48:40.858995 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:40.858983    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:48:41.005611 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:41.005573    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8" event={"ID":"3da5e6df-b15c-4209-9cec-624599f6713d","Type":"ContainerStarted","Data":"3a010c030ef32387455a5a37809d0239a3978b475bf62b6554cbfffa65afd7c9"}
Nov 02 01:48:41.006107 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:41.005641    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8" event={"ID":"3da5e6df-b15c-4209-9cec-624599f6713d","Type":"ContainerStarted","Data":"7be4184ab26a1ae9563df26d4a1405a5a594551091a2ae84681b817b96fae0c3"}
Nov 02 01:48:41.007866 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:41.007837    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz" event={"ID":"47285028-a487-437d-8cd9-63221ffca4d6","Type":"ContainerStarted","Data":"22dbaa0aa4fc3f86ba306177aaa06e8d827b0b6fe937b505a40ae6b0873de394"}
Nov 02 01:48:41.008019 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:41.007890    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz" event={"ID":"47285028-a487-437d-8cd9-63221ffca4d6","Type":"ContainerStarted","Data":"f6af77db99e1f05f4bcc4930d952341303ac736390ac76833dda9c69238a8599"}
Nov 02 01:48:41.139849 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:41.137510    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8" podStartSLOduration=2.137490288 podStartE2EDuration="2.137490288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:41.128013321 +0000 UTC m=+1386.031695460" watchObservedRunningTime="2026-11-02 01:48:41.137490288 +0000 UTC m=+1386.041172429"
Nov 02 01:48:41.179303 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:48:41.178781    5639 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7048-4e00a9\" is forbidden: User \"system:node:ip-10-0-136-53.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-53.us-west-1.compute.internal' and this object" podUID="8ed19135-0549-455d-aacd-fbdbc1cc8858" pod="opendatahub/test-curl-7048-4e00a9"
Nov 02 01:48:41.303065 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:41.302995    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz" podStartSLOduration=2.302975947 podStartE2EDuration="2.302975947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:41.298136588 +0000 UTC m=+1386.201818702" watchObservedRunningTime="2026-11-02 01:48:41.302975947 +0000 UTC m=+1386.206658062"
Nov 02 01:48:41.349079 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:48:41.349041    5639 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7048-4e00a9\" is forbidden: User \"system:node:ip-10-0-136-53.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-53.us-west-1.compute.internal' and this object" podUID="8ed19135-0549-455d-aacd-fbdbc1cc8858" pod="opendatahub/test-curl-7048-4e00a9"
Nov 02 01:48:41.477363 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:41.474736    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:41.477363 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:41.474929    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 02 01:48:41.477363 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:41.474979    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz" podUID="47285028-a487-437d-8cd9-63221ffca4d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 02 01:48:42.000305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:41.993767    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:42.000305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:41.993939    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 02 01:48:42.000305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:41.994003    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8" podUID="3da5e6df-b15c-4209-9cec-624599f6713d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 02 01:48:42.482313 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:42.475396    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 02 01:48:42.482313 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:42.475473    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz" podUID="47285028-a487-437d-8cd9-63221ffca4d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 02 01:48:42.524648 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:42.524623    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7048-4e00a9"
Nov 02 01:48:42.628689 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:42.628603    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ed19135-0549-455d-aacd-fbdbc1cc8858-kube-api-access-vt924\" (UniqueName: \"kubernetes.io/projected/8ed19135-0549-455d-aacd-fbdbc1cc8858-kube-api-access-vt924\") pod \"8ed19135-0549-455d-aacd-fbdbc1cc8858\" (UID: \"8ed19135-0549-455d-aacd-fbdbc1cc8858\") "
Nov 02 01:48:42.631166 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:42.631140    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed19135-0549-455d-aacd-fbdbc1cc8858-kube-api-access-vt924" pod "8ed19135-0549-455d-aacd-fbdbc1cc8858" (UID: "8ed19135-0549-455d-aacd-fbdbc1cc8858"). InnerVolumeSpecName "kube-api-access-vt924". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:42.730837 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:42.730797    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt924\" (UniqueName: \"kubernetes.io/projected/8ed19135-0549-455d-aacd-fbdbc1cc8858-kube-api-access-vt924\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:43.007884 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:43.007782    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 02 01:48:43.008158 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:43.008128    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8" podUID="3da5e6df-b15c-4209-9cec-624599f6713d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 02 01:48:43.029571 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:43.029539    5639 generic.go:356] "Generic (PLEG): container finished" podID="8ed19135-0549-455d-aacd-fbdbc1cc8858" containerID="d36428fc5412f1a9d000cb958c50c5a06514ef15503a5dc384a3070c91b06671" exitCode=137
Nov 02 01:48:43.029779 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:43.029588    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7048-4e00a9"
Nov 02 01:48:43.029867 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:43.029621    5639 scope.go:122] "RemoveContainer" containerID="d36428fc5412f1a9d000cb958c50c5a06514ef15503a5dc384a3070c91b06671"
Nov 02 01:48:43.045543 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:43.045434    5639 scope.go:122] "RemoveContainer" containerID="d36428fc5412f1a9d000cb958c50c5a06514ef15503a5dc384a3070c91b06671"
Nov 02 01:48:43.045801 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:48:43.045778    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d36428fc5412f1a9d000cb958c50c5a06514ef15503a5dc384a3070c91b06671\": container with ID starting with d36428fc5412f1a9d000cb958c50c5a06514ef15503a5dc384a3070c91b06671 not found: ID does not exist" containerID="d36428fc5412f1a9d000cb958c50c5a06514ef15503a5dc384a3070c91b06671"
Nov 02 01:48:43.045871 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:43.045810    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d36428fc5412f1a9d000cb958c50c5a06514ef15503a5dc384a3070c91b06671"} err="failed to get container status \"d36428fc5412f1a9d000cb958c50c5a06514ef15503a5dc384a3070c91b06671\": rpc error: code = NotFound desc = could not find container \"d36428fc5412f1a9d000cb958c50c5a06514ef15503a5dc384a3070c91b06671\": container with ID starting with d36428fc5412f1a9d000cb958c50c5a06514ef15503a5dc384a3070c91b06671 not found: ID does not exist"
Nov 02 01:48:43.099307 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:48:43.098548    5639 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7048-4e00a9\" is forbidden: User \"system:node:ip-10-0-136-53.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-53.us-west-1.compute.internal' and this object" podUID="8ed19135-0549-455d-aacd-fbdbc1cc8858" pod="opendatahub/test-curl-7048-4e00a9"
Nov 02 01:48:43.221557 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:43.221521    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ed19135-0549-455d-aacd-fbdbc1cc8858" path="/var/lib/kubelet/pods/8ed19135-0549-455d-aacd-fbdbc1cc8858/volumes"
Nov 02 01:48:43.480307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:43.471779    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 02 01:48:43.480307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:43.471865    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz" podUID="47285028-a487-437d-8cd9-63221ffca4d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 02 01:48:43.994164 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:43.994127    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 02 01:48:43.994810 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:43.994777    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8" podUID="3da5e6df-b15c-4209-9cec-624599f6713d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 02 01:48:44.482313 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:44.477774    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 02 01:48:44.482313 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:44.477842    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz" podUID="47285028-a487-437d-8cd9-63221ffca4d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 02 01:48:44.999027 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:44.998986    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 02 01:48:44.999676 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:44.999643    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8" podUID="3da5e6df-b15c-4209-9cec-624599f6713d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 02 01:48:45.489186 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:45.489147    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 02 01:48:45.489499 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:45.489469    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz" podUID="47285028-a487-437d-8cd9-63221ffca4d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 02 01:48:45.996480 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:45.996447    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 02 01:48:45.996913 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:45.996884    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8" podUID="3da5e6df-b15c-4209-9cec-624599f6713d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 02 01:48:46.479308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:46.476718    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 02 01:48:46.479308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:46.476783    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz" podUID="47285028-a487-437d-8cd9-63221ffca4d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 02 01:48:46.992690 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:46.992650    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 02 01:48:46.992923 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:46.992726    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8" podUID="3da5e6df-b15c-4209-9cec-624599f6713d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 02 01:48:47.051056 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:47.051014    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a4134a0-s666n"]
Nov 02 01:48:47.102318 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:47.102255    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a4134a0-s666n"]
Nov 02 01:48:47.227176 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:47.227127    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="105ca4e7-8651-4327-98e1-34a27970d9b2" path="/var/lib/kubelet/pods/105ca4e7-8651-4327-98e1-34a27970d9b2/volumes"
Nov 02 01:48:47.468231 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:47.468192    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 02 01:48:47.468440 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:47.468262    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz" podUID="47285028-a487-437d-8cd9-63221ffca4d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 02 01:48:48.186481 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:48.186416    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:48.187110 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:48.187090    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:48.188001 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:48.187979    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:48:48.552510 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:48.552431    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:48.562349 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:48.562261    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:48.565476 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:48.565451    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:48:50.267669 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.267628    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"]
Nov 02 01:48:50.273989 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.273960    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.439331 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.436659    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2821b375-openshift-default-dockercfg-lq2nh\""
Nov 02 01:48:50.445544 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.445511    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-workload-socket\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.445882 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.445860    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istiod-ca-cert\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.446053 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.446038    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-envoy\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.446431 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.446406    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-credential-socket\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.446733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.446715    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-data\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.446863 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.446845    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-token\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.447022 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.447006    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-podinfo\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.447189 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.447170    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln6w8\" (UniqueName: \"kubernetes.io/projected/8b71baa1-9d31-48de-a3ec-379f6f5eae37-kube-api-access-ln6w8\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.447382 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.447354    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-workload-certs\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.455670 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.455639    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"]
Nov 02 01:48:50.551616 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.551515    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istiod-ca-cert\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.553225 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.553200    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-envoy\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.560843 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.560796    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-credential-socket\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.561760 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.561730    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-data\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.562352 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.562333    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-token\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.568545 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.568507    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-podinfo\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.569661 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.569634    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln6w8\" (UniqueName: \"kubernetes.io/projected/8b71baa1-9d31-48de-a3ec-379f6f5eae37-kube-api-access-ln6w8\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.570318 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.570253    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-workload-certs\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.576789 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.576708    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-workload-socket\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.577731 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.561500    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-credential-socket\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.577963 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.570759    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-workload-certs\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.580118 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.553061    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istiod-ca-cert\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.580328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.577504    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-workload-socket\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.580477 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.562243    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-data\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.645106 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.645071    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-envoy\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.674187 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.673721    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-podinfo\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.680739 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.680705    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-token\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.683779 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.683731    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln6w8\" (UniqueName: \"kubernetes.io/projected/8b71baa1-9d31-48de-a3ec-379f6f5eae37-kube-api-access-ln6w8\") pod \"e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") " pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:50.889017 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:50.888979    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:51.032673 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:51.032631    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8151ea22-xkh47"]
Nov 02 01:48:51.039833 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:51.039795    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8151ea22-xkh47"
Nov 02 01:48:51.092652 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:51.092490    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:48:51.092652 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:51.092538    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:48:51.092652 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:51.092511    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:48:51.092652 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:51.092600    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8151ea22-xkh47"]
Nov 02 01:48:51.092652 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:51.092619    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:48:51.111362 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:51.111329    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2baea30f-f4e8-4c30-9754-0a1192ad0b26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8151ea22-xkh47\" (UID: \"2baea30f-f4e8-4c30-9754-0a1192ad0b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8151ea22-xkh47"
Nov 02 01:48:51.111592 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:51.111567    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjmcb\" (UniqueName: \"kubernetes.io/projected/2baea30f-f4e8-4c30-9754-0a1192ad0b26-kube-api-access-bjmcb\") pod \"maas-api-revoke-sub-8151ea22-xkh47\" (UID: \"2baea30f-f4e8-4c30-9754-0a1192ad0b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8151ea22-xkh47"
Nov 02 01:48:51.111739 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:51.111721    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2baea30f-f4e8-4c30-9754-0a1192ad0b26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8151ea22-xkh47\" (UID: \"2baea30f-f4e8-4c30-9754-0a1192ad0b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8151ea22-xkh47"
Nov 02 01:48:51.214192 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:51.214055    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjmcb\" (UniqueName: \"kubernetes.io/projected/2baea30f-f4e8-4c30-9754-0a1192ad0b26-kube-api-access-bjmcb\") pod \"maas-api-revoke-sub-8151ea22-xkh47\" (UID: \"2baea30f-f4e8-4c30-9754-0a1192ad0b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8151ea22-xkh47"
Nov 02 01:48:51.214509 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:51.214194    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2baea30f-f4e8-4c30-9754-0a1192ad0b26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8151ea22-xkh47\" (UID: \"2baea30f-f4e8-4c30-9754-0a1192ad0b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8151ea22-xkh47"
Nov 02 01:48:51.214686 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:51.214639    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2baea30f-f4e8-4c30-9754-0a1192ad0b26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8151ea22-xkh47\" (UID: \"2baea30f-f4e8-4c30-9754-0a1192ad0b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8151ea22-xkh47"
Nov 02 01:48:51.215700 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:51.215680    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2baea30f-f4e8-4c30-9754-0a1192ad0b26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8151ea22-xkh47\" (UID: \"2baea30f-f4e8-4c30-9754-0a1192ad0b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8151ea22-xkh47"
Nov 02 01:48:51.217385 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:51.217363    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2baea30f-f4e8-4c30-9754-0a1192ad0b26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8151ea22-xkh47\" (UID: \"2baea30f-f4e8-4c30-9754-0a1192ad0b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8151ea22-xkh47"
Nov 02 01:48:51.276513 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:51.276467    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjmcb\" (UniqueName: \"kubernetes.io/projected/2baea30f-f4e8-4c30-9754-0a1192ad0b26-kube-api-access-bjmcb\") pod \"maas-api-revoke-sub-8151ea22-xkh47\" (UID: \"2baea30f-f4e8-4c30-9754-0a1192ad0b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8151ea22-xkh47"
Nov 02 01:48:51.364954 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:51.364918    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8151ea22-xkh47"
Nov 02 01:48:51.394788 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:51.383349    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"]
Nov 02 01:48:51.394788 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:48:51.388941    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b71baa1_9d31_48de_a3ec_379f6f5eae37.slice/crio-6544d64e61a4d8e32cd392d4da5f1a4d627bb1d5ccf687044483fb1f9648a824 WatchSource:0}: Error finding container 6544d64e61a4d8e32cd392d4da5f1a4d627bb1d5ccf687044483fb1f9648a824: Status 404 returned error can't find the container with id 6544d64e61a4d8e32cd392d4da5f1a4d627bb1d5ccf687044483fb1f9648a824
Nov 02 01:48:51.413583 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:51.413529    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:48:51.413784 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:51.413625    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:48:51.413784 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:51.413669    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:48:52.004428 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:52.004391    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8151ea22-xkh47"]
Nov 02 01:48:52.288322 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:52.287891    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5" event={"ID":"8b71baa1-9d31-48de-a3ec-379f6f5eae37","Type":"ContainerStarted","Data":"5107a59619749632b90e634b66f0b7ba6535efb394ae63ff6752b6f76fc7bbe0"}
Nov 02 01:48:52.288322 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:52.287950    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5" event={"ID":"8b71baa1-9d31-48de-a3ec-379f6f5eae37","Type":"ContainerStarted","Data":"6544d64e61a4d8e32cd392d4da5f1a4d627bb1d5ccf687044483fb1f9648a824"}
Nov 02 01:48:52.291862 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:52.291825    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8151ea22-xkh47" event={"ID":"2baea30f-f4e8-4c30-9754-0a1192ad0b26","Type":"ContainerStarted","Data":"18a017697a4910fd40a77f2a079acdd4594555e31c60c2f31b63b77682777a32"}
Nov 02 01:48:52.480433 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:52.479304    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5" podStartSLOduration=2.479258922 podStartE2EDuration="2.479258922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:52.456862351 +0000 UTC m=+1397.360544477" watchObservedRunningTime="2026-11-02 01:48:52.479258922 +0000 UTC m=+1397.382941039"
Nov 02 01:48:52.895306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:52.889952    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:52.904402 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:52.904372    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:53.302004 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:53.301915    5639 generic.go:356] "Generic (PLEG): container finished" podID="2baea30f-f4e8-4c30-9754-0a1192ad0b26" containerID="c5279e59befdebb76f7a6198bcb4efcc288f8728d5e7aca87757353792db2271" exitCode=0
Nov 02 01:48:53.302495 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:53.302002    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8151ea22-xkh47" event={"ID":"2baea30f-f4e8-4c30-9754-0a1192ad0b26","Type":"ContainerDied","Data":"c5279e59befdebb76f7a6198bcb4efcc288f8728d5e7aca87757353792db2271"}
Nov 02 01:48:53.302495 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:53.302372    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:53.303626 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:53.303603    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:48:54.504009 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:54.503979    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8151ea22-xkh47"
Nov 02 01:48:54.572215 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:54.572170    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2baea30f-f4e8-4c30-9754-0a1192ad0b26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2baea30f-f4e8-4c30-9754-0a1192ad0b26-maas-api-xks-ca\") pod \"2baea30f-f4e8-4c30-9754-0a1192ad0b26\" (UID: \"2baea30f-f4e8-4c30-9754-0a1192ad0b26\") "
Nov 02 01:48:54.572598 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:54.572578    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2baea30f-f4e8-4c30-9754-0a1192ad0b26-kube-api-access-bjmcb\" (UniqueName: \"kubernetes.io/projected/2baea30f-f4e8-4c30-9754-0a1192ad0b26-kube-api-access-bjmcb\") pod \"2baea30f-f4e8-4c30-9754-0a1192ad0b26\" (UID: \"2baea30f-f4e8-4c30-9754-0a1192ad0b26\") "
Nov 02 01:48:54.572684 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:54.572622    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2baea30f-f4e8-4c30-9754-0a1192ad0b26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2baea30f-f4e8-4c30-9754-0a1192ad0b26-maas-api-openshift-ca\") pod \"2baea30f-f4e8-4c30-9754-0a1192ad0b26\" (UID: \"2baea30f-f4e8-4c30-9754-0a1192ad0b26\") "
Nov 02 01:48:54.573059 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:54.573035    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2baea30f-f4e8-4c30-9754-0a1192ad0b26-maas-api-openshift-ca" pod "2baea30f-f4e8-4c30-9754-0a1192ad0b26" (UID: "2baea30f-f4e8-4c30-9754-0a1192ad0b26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:54.573269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:54.573205    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2baea30f-f4e8-4c30-9754-0a1192ad0b26-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:54.574753 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:54.574721    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2baea30f-f4e8-4c30-9754-0a1192ad0b26-maas-api-xks-ca" pod "2baea30f-f4e8-4c30-9754-0a1192ad0b26" (UID: "2baea30f-f4e8-4c30-9754-0a1192ad0b26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:54.574960 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:54.574939    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2baea30f-f4e8-4c30-9754-0a1192ad0b26-kube-api-access-bjmcb" pod "2baea30f-f4e8-4c30-9754-0a1192ad0b26" (UID: "2baea30f-f4e8-4c30-9754-0a1192ad0b26"). InnerVolumeSpecName "kube-api-access-bjmcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:54.674562 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:54.674524    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjmcb\" (UniqueName: \"kubernetes.io/projected/2baea30f-f4e8-4c30-9754-0a1192ad0b26-kube-api-access-bjmcb\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:54.674562 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:54.674564    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2baea30f-f4e8-4c30-9754-0a1192ad0b26-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:55.336981 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.336954    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8151ea22-xkh47"
Nov 02 01:48:55.337181 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.336986    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8151ea22-xkh47" event={"ID":"2baea30f-f4e8-4c30-9754-0a1192ad0b26","Type":"ContainerDied","Data":"18a017697a4910fd40a77f2a079acdd4594555e31c60c2f31b63b77682777a32"}
Nov 02 01:48:55.337181 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.337044    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18a017697a4910fd40a77f2a079acdd4594555e31c60c2f31b63b77682777a32"
Nov 02 01:48:55.350614 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.350583    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"]
Nov 02 01:48:55.351265 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.351249    5639 cpu_manager.go:414] "containerMap: removing container" podUID="2baea30f-f4e8-4c30-9754-0a1192ad0b26" containerName="revoke-keys"
Nov 02 01:48:55.351371 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.351267    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2baea30f-f4e8-4c30-9754-0a1192ad0b26" containerName="revoke-keys"
Nov 02 01:48:55.351442 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.351431    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="2baea30f-f4e8-4c30-9754-0a1192ad0b26" containerName="revoke-keys"
Nov 02 01:48:55.355101 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.355081    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.451656 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.451625    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"]
Nov 02 01:48:55.487235 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.487134    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.487235 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.487209    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.487546 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.487337    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.487546 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.487406    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.487663 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.487546    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94mkq\" (UniqueName: \"kubernetes.io/projected/b4bfc70f-fa52-4876-9699-56040fd06e57-kube-api-access-94mkq\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.487663 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.487608    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.487771 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.487744    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.487823 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.487782    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4bfc70f-fa52-4876-9699-56040fd06e57-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.487883 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.487873    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.589710 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.589675    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.590168 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.589725    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4bfc70f-fa52-4876-9699-56040fd06e57-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.590168 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.589838    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.590168 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.589913    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.590168 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.590004    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.590168 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.590049    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.590168 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.590096    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.590168 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.590137    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.591206 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.590179    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94mkq\" (UniqueName: \"kubernetes.io/projected/b4bfc70f-fa52-4876-9699-56040fd06e57-kube-api-access-94mkq\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.591206 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.590191    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.591206 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.590405    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.591206 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.590544    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.591206 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.590700    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4bfc70f-fa52-4876-9699-56040fd06e57-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.591206 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.590762    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.592796 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.592768    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.592902 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.592872    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.697168 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.694067    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94mkq\" (UniqueName: \"kubernetes.io/projected/b4bfc70f-fa52-4876-9699-56040fd06e57-kube-api-access-94mkq\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.738435 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.734894    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:55.970825 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:55.970795    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:56.496661 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:56.496629    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"]
Nov 02 01:48:56.496795 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:48:56.496766    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4bfc70f_fa52_4876_9699_56040fd06e57.slice/crio-5327dfa7f0df0a92227c69830f2483e6f54bb2c3f7edee47d46d4f7ac1cbeffc WatchSource:0}: Error finding container 5327dfa7f0df0a92227c69830f2483e6f54bb2c3f7edee47d46d4f7ac1cbeffc: Status 404 returned error can't find the container with id 5327dfa7f0df0a92227c69830f2483e6f54bb2c3f7edee47d46d4f7ac1cbeffc
Nov 02 01:48:56.499896 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:56.499858    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:48:56.499975 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:56.499948    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:48:56.500012 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:56.499977    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:48:57.422739 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:48:57.422699    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:48:57.468036 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:57.468000    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz" event={"ID":"b4bfc70f-fa52-4876-9699-56040fd06e57","Type":"ContainerStarted","Data":"08b6aea114889adf04b7352833b29ef17ea0dc8d1b662b031adf295a951c9a5e"}
Nov 02 01:48:57.468208 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:57.468094    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz" event={"ID":"b4bfc70f-fa52-4876-9699-56040fd06e57","Type":"ContainerStarted","Data":"5327dfa7f0df0a92227c69830f2483e6f54bb2c3f7edee47d46d4f7ac1cbeffc"}
Nov 02 01:48:57.595534 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:57.595481    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz" podStartSLOduration=2.595466639 podStartE2EDuration="2.595466639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:57.593005168 +0000 UTC m=+1402.496687299" watchObservedRunningTime="2026-11-02 01:48:57.595466639 +0000 UTC m=+1402.499148753"
Nov 02 01:48:57.973314 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:57.971689    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:48:57.973529 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:57.973435    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 02 01:48:57.973529 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:57.973502    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz" podUID="b4bfc70f-fa52-4876-9699-56040fd06e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 02 01:48:58.986108 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:58.986016    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 02 01:48:58.986108 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:58.986092    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz" podUID="b4bfc70f-fa52-4876-9699-56040fd06e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 02 01:48:59.983429 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:59.983389    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 02 01:48:59.983720 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:48:59.983692    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz" podUID="b4bfc70f-fa52-4876-9699-56040fd06e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 02 01:49:00.982323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:00.975370    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 02 01:49:00.982323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:00.975441    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz" podUID="b4bfc70f-fa52-4876-9699-56040fd06e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 02 01:49:01.796305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:01.795841    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a330ba00-whkqc"]
Nov 02 01:49:01.800463 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:01.800144    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a330ba00-whkqc"
Nov 02 01:49:01.899387 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:01.899350    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:49:01.900087 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:01.900060    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:49:01.900216 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:01.900209    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:49:01.900611 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:01.900588    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a330ba00-whkqc"]
Nov 02 01:49:01.902079 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:01.902048    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl44l\" (UniqueName: \"kubernetes.io/projected/26bb5d7f-26f6-4a7f-9dba-e5b3239b536c-kube-api-access-tl44l\") pod \"maas-api-revoke-sub-a330ba00-whkqc\" (UID: \"26bb5d7f-26f6-4a7f-9dba-e5b3239b536c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a330ba00-whkqc"
Nov 02 01:49:01.902170 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:01.902112    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26bb5d7f-26f6-4a7f-9dba-e5b3239b536c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a330ba00-whkqc\" (UID: \"26bb5d7f-26f6-4a7f-9dba-e5b3239b536c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a330ba00-whkqc"
Nov 02 01:49:01.902250 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:01.902221    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26bb5d7f-26f6-4a7f-9dba-e5b3239b536c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a330ba00-whkqc\" (UID: \"26bb5d7f-26f6-4a7f-9dba-e5b3239b536c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a330ba00-whkqc"
Nov 02 01:49:01.906160 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:01.906130    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:49:01.987382 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:01.977987    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 02 01:49:01.987887 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:01.987443    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz" podUID="b4bfc70f-fa52-4876-9699-56040fd06e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 02 01:49:02.004450 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:02.004413    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl44l\" (UniqueName: \"kubernetes.io/projected/26bb5d7f-26f6-4a7f-9dba-e5b3239b536c-kube-api-access-tl44l\") pod \"maas-api-revoke-sub-a330ba00-whkqc\" (UID: \"26bb5d7f-26f6-4a7f-9dba-e5b3239b536c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a330ba00-whkqc"
Nov 02 01:49:02.004656 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:02.004462    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26bb5d7f-26f6-4a7f-9dba-e5b3239b536c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a330ba00-whkqc\" (UID: \"26bb5d7f-26f6-4a7f-9dba-e5b3239b536c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a330ba00-whkqc"
Nov 02 01:49:02.004656 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:02.004609    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26bb5d7f-26f6-4a7f-9dba-e5b3239b536c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a330ba00-whkqc\" (UID: \"26bb5d7f-26f6-4a7f-9dba-e5b3239b536c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a330ba00-whkqc"
Nov 02 01:49:02.005052 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:02.005031    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26bb5d7f-26f6-4a7f-9dba-e5b3239b536c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a330ba00-whkqc\" (UID: \"26bb5d7f-26f6-4a7f-9dba-e5b3239b536c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a330ba00-whkqc"
Nov 02 01:49:02.007223 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:02.007204    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26bb5d7f-26f6-4a7f-9dba-e5b3239b536c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a330ba00-whkqc\" (UID: \"26bb5d7f-26f6-4a7f-9dba-e5b3239b536c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a330ba00-whkqc"
Nov 02 01:49:02.087122 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:02.087078    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl44l\" (UniqueName: \"kubernetes.io/projected/26bb5d7f-26f6-4a7f-9dba-e5b3239b536c-kube-api-access-tl44l\") pod \"maas-api-revoke-sub-a330ba00-whkqc\" (UID: \"26bb5d7f-26f6-4a7f-9dba-e5b3239b536c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a330ba00-whkqc"
Nov 02 01:49:02.117310 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:02.116054    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a330ba00-whkqc"
Nov 02 01:49:02.532622 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:49:02.531556    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26bb5d7f_26f6_4a7f_9dba_e5b3239b536c.slice/crio-ba0b04a56ea4183dd4ce910e6e123309ad5e874fa6dca4460f9c67ae5fda276c WatchSource:0}: Error finding container ba0b04a56ea4183dd4ce910e6e123309ad5e874fa6dca4460f9c67ae5fda276c: Status 404 returned error can't find the container with id ba0b04a56ea4183dd4ce910e6e123309ad5e874fa6dca4460f9c67ae5fda276c
Nov 02 01:49:02.533447 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:02.533421    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a330ba00-whkqc"]
Nov 02 01:49:02.978060 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:02.978026    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 02 01:49:02.978382 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:02.978348    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz" podUID="b4bfc70f-fa52-4876-9699-56040fd06e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 02 01:49:03.540064 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:03.540028    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a330ba00-whkqc" event={"ID":"26bb5d7f-26f6-4a7f-9dba-e5b3239b536c","Type":"ContainerStarted","Data":"f24f14cc5087e85696f02ac7e95cc8a57a074142ca56b09c5bb7508f04be35c2"}
Nov 02 01:49:03.540527 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:03.540086    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a330ba00-whkqc" event={"ID":"26bb5d7f-26f6-4a7f-9dba-e5b3239b536c","Type":"ContainerStarted","Data":"ba0b04a56ea4183dd4ce910e6e123309ad5e874fa6dca4460f9c67ae5fda276c"}
Nov 02 01:49:03.630328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:03.630241    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:49:03.661414 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:03.661360    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a330ba00-whkqc" podStartSLOduration=2.661344941 podStartE2EDuration="2.661344941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:03.65971588 +0000 UTC m=+1408.563397997" watchObservedRunningTime="2026-11-02 01:49:03.661344941 +0000 UTC m=+1408.565027055"
Nov 02 01:49:04.553410 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:04.553369    5639 generic.go:356] "Generic (PLEG): container finished" podID="26bb5d7f-26f6-4a7f-9dba-e5b3239b536c" containerID="f24f14cc5087e85696f02ac7e95cc8a57a074142ca56b09c5bb7508f04be35c2" exitCode=0
Nov 02 01:49:04.553979 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:04.553580    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a330ba00-whkqc" event={"ID":"26bb5d7f-26f6-4a7f-9dba-e5b3239b536c","Type":"ContainerDied","Data":"f24f14cc5087e85696f02ac7e95cc8a57a074142ca56b09c5bb7508f04be35c2"}
Nov 02 01:49:05.046315 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:05.045788    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:49:05.046315 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:05.045899    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz" podUID="b4bfc70f-fa52-4876-9699-56040fd06e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:05.656434 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:05.656382    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:49:05.886454 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:05.886411    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:06.009494 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:06.009397    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:49:06.079306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:06.075456    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:49:06.079306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:06.075537    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz" podUID="b4bfc70f-fa52-4876-9699-56040fd06e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:06.202531 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:06.201662    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:06.279768 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:06.279673    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:06.312487 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:06.311386    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:49:06.345467 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:06.345377    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:06.696126 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:06.693837    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:06.696126 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:06.694160    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:49:06.696126 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:06.694836    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:49:06.740646 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:06.740601    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:07.117641 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:07.097538    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:49:07.117641 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:07.098320    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:49:07.117641 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:07.102379    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:49:07.117641 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:07.102802    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:49:07.278481 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:07.146762    5639 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:49:07.649422 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:07.649374    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4" podUID="a8596379-fe63-4152-8569-1b16a8f75d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:07.691760 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:07.223966    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": context deadline exceeded"
Nov 02 01:49:07.702050 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:07.282747    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:07.735824 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:07.328705    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:07.774578 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:07.736473    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 02 01:49:07.774578 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:07.223804    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:49:07.774578 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:07.737268    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz" podUID="b4bfc70f-fa52-4876-9699-56040fd06e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:07.774578 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:07.737504    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:07.774578 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:07.190343    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:49:07.837078 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:07.832720    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:08.080364 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.066452    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a330ba00-whkqc"
Nov 02 01:49:08.114874 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.114830    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:08.140455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.126226    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:49:08.144889 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.144858    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:49:08.152478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.152450    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:49:08.152808 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.152792    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:49:08.152979 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.152965    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:49:08.153159 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.153135    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:49:08.168706 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.168663    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:08.181323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.180345    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26bb5d7f-26f6-4a7f-9dba-e5b3239b536c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26bb5d7f-26f6-4a7f-9dba-e5b3239b536c-maas-api-openshift-ca\") pod \"26bb5d7f-26f6-4a7f-9dba-e5b3239b536c\" (UID: \"26bb5d7f-26f6-4a7f-9dba-e5b3239b536c\") "
Nov 02 01:49:08.181323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.180499    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26bb5d7f-26f6-4a7f-9dba-e5b3239b536c-kube-api-access-tl44l\" (UniqueName: \"kubernetes.io/projected/26bb5d7f-26f6-4a7f-9dba-e5b3239b536c-kube-api-access-tl44l\") pod \"26bb5d7f-26f6-4a7f-9dba-e5b3239b536c\" (UID: \"26bb5d7f-26f6-4a7f-9dba-e5b3239b536c\") "
Nov 02 01:49:08.181323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.180572    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26bb5d7f-26f6-4a7f-9dba-e5b3239b536c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26bb5d7f-26f6-4a7f-9dba-e5b3239b536c-maas-api-xks-ca\") pod \"26bb5d7f-26f6-4a7f-9dba-e5b3239b536c\" (UID: \"26bb5d7f-26f6-4a7f-9dba-e5b3239b536c\") "
Nov 02 01:49:08.194343 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.188385    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:49:08.194343 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.188448    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp" podUID="eb685866-3bb8-43ae-85f9-17cea849a9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:08.214560 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.214514    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26bb5d7f-26f6-4a7f-9dba-e5b3239b536c-maas-api-openshift-ca" pod "26bb5d7f-26f6-4a7f-9dba-e5b3239b536c" (UID: "26bb5d7f-26f6-4a7f-9dba-e5b3239b536c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:08.301500 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.242527    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26bb5d7f-26f6-4a7f-9dba-e5b3239b536c-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:08.420583 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.420475    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26bb5d7f-26f6-4a7f-9dba-e5b3239b536c-kube-api-access-tl44l" pod "26bb5d7f-26f6-4a7f-9dba-e5b3239b536c" (UID: "26bb5d7f-26f6-4a7f-9dba-e5b3239b536c"). InnerVolumeSpecName "kube-api-access-tl44l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:08.472310 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.457191    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl44l\" (UniqueName: \"kubernetes.io/projected/26bb5d7f-26f6-4a7f-9dba-e5b3239b536c-kube-api-access-tl44l\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:08.484518 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.484475    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26bb5d7f-26f6-4a7f-9dba-e5b3239b536c-maas-api-xks-ca" pod "26bb5d7f-26f6-4a7f-9dba-e5b3239b536c" (UID: "26bb5d7f-26f6-4a7f-9dba-e5b3239b536c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:08.590712 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.590673    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26bb5d7f-26f6-4a7f-9dba-e5b3239b536c-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:08.878155 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.878087    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": context deadline exceeded"
Nov 02 01:49:08.907371 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.890457    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:49:08.907914 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.890493    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 02 01:49:08.908368 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.900406    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:08.912652 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.900443    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 02 01:49:08.913091 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.900562    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:09.009310 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:08.900585    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:49:09.024335 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:09.010433    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:49:09.024335 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:09.018431    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:49:09.024335 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:09.018486    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz" podUID="b4bfc70f-fa52-4876-9699-56040fd06e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:09.070020 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:09.041477    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/health\": context deadline exceeded"
Nov 02 01:49:09.173361 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:09.155099    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:49:09.173361 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:09.155863    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:49:09.449855 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:09.449759    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:09.919142 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:09.919095    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:10.106867 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:10.106803    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:49:10.107618 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:10.107573    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 02 01:49:10.113943 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:10.109247    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Nov 02 01:49:10.114910 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:10.111069    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:10.127489 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:10.127449    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:10.315880 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:10.156412    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:10.335304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:10.173097    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:10.335304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:10.302798    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:10.335304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:10.303027    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:49:10.335304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:10.330700    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz" podUID="b4bfc70f-fa52-4876-9699-56040fd06e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:10.600233 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:10.600185    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a330ba00-whkqc" event={"ID":"26bb5d7f-26f6-4a7f-9dba-e5b3239b536c","Type":"ContainerDied","Data":"ba0b04a56ea4183dd4ce910e6e123309ad5e874fa6dca4460f9c67ae5fda276c"}
Nov 02 01:49:10.600609 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:10.600571    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba0b04a56ea4183dd4ce910e6e123309ad5e874fa6dca4460f9c67ae5fda276c"
Nov 02 01:49:10.626653 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:10.626609    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a330ba00-whkqc"
Nov 02 01:49:10.715662 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:10.692662    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:49:10.715662 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:10.692733    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb" podUID="fb887678-0155-4d81-be15-861a7e1f8ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:10.716330 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:10.716269    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:10.752761 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:10.752729    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:49:10.835532 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:10.821925    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:49:10.847318 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:10.845782    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw" podUID="cbdac3f5-20af-4282-b567-66a4d880c3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:11.117310 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:11.107444    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:49:11.157311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:11.156993    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:11.169127 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:11.158904    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:11.221676 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:11.221612    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh" podUID="6b06646f-8efd-4ebc-9b9c-ff8f9734da72" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:11.307354 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:11.227336    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:11.321371 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:11.319364    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:11.509369 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:11.487991    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:49:11.509369 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:11.488111    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:49:11.621154 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:11.621088    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 01:49:11.623909 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:11.623854    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:11.627831 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:11.627803    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:49:11.645630 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:11.645588    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:49:11.661487 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:11.661437    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz" podUID="b4bfc70f-fa52-4876-9699-56040fd06e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:11.940301 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:11.938391    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:49:12.172309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:12.166727    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:12.172309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:12.167159    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 02 01:49:12.313206 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:12.313108    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:49:12.365351 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:12.318705    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:12.380617 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:12.349753    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:12.635654 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:12.635603    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:49:12.705147 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:12.682419    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:49:12.705147 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:12.682481    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz" podUID="b4bfc70f-fa52-4876-9699-56040fd06e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:12.791370 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:12.791328    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:12.959374 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:12.954031    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:12.973588 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:12.973471    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:13.172724 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:13.172043    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:13.174444 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:13.173236    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 02 01:49:13.331630 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:13.316912    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:49:13.331630 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:13.320202    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:49:13.341323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:13.341273    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:49:13.342711 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:13.342691    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 01:49:13.379479 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:13.379439    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:13.390414 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:13.390375    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:13.390997 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:13.390978    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:49:13.431833 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:13.422740    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:49:13.447093 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:13.447064    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:49:13.458811 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:13.458783    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:49:13.461940 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:13.461916    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 01:49:13.480017 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:13.479991    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:49:18.748751 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:18.748708    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f154566-b8n47"]
Nov 02 01:49:18.749499 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:18.749480    5639 cpu_manager.go:414] "containerMap: removing container" podUID="26bb5d7f-26f6-4a7f-9dba-e5b3239b536c" containerName="revoke-keys"
Nov 02 01:49:18.749499 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:18.749499    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26bb5d7f-26f6-4a7f-9dba-e5b3239b536c" containerName="revoke-keys"
Nov 02 01:49:18.749645 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:18.749635    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="26bb5d7f-26f6-4a7f-9dba-e5b3239b536c" containerName="revoke-keys"
Nov 02 01:49:18.757065 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:18.757042    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f154566-b8n47"
Nov 02 01:49:18.783054 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:18.783018    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkpb2\" (UniqueName: \"kubernetes.io/projected/080f5dc7-c78f-4c7f-ab59-e7e05988b487-kube-api-access-zkpb2\") pod \"maas-api-revoke-sub-9f154566-b8n47\" (UID: \"080f5dc7-c78f-4c7f-ab59-e7e05988b487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f154566-b8n47"
Nov 02 01:49:18.783247 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:18.783085    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080f5dc7-c78f-4c7f-ab59-e7e05988b487-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f154566-b8n47\" (UID: \"080f5dc7-c78f-4c7f-ab59-e7e05988b487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f154566-b8n47"
Nov 02 01:49:18.783247 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:18.783225    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080f5dc7-c78f-4c7f-ab59-e7e05988b487-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f154566-b8n47\" (UID: \"080f5dc7-c78f-4c7f-ab59-e7e05988b487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f154566-b8n47"
Nov 02 01:49:18.811926 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:18.811883    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f154566-b8n47"]
Nov 02 01:49:18.855657 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:18.855614    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:49:18.855927 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:18.855694    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:49:18.855927 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:18.855781    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:49:18.858782 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:18.858753    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:49:18.885215 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:18.885176    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkpb2\" (UniqueName: \"kubernetes.io/projected/080f5dc7-c78f-4c7f-ab59-e7e05988b487-kube-api-access-zkpb2\") pod \"maas-api-revoke-sub-9f154566-b8n47\" (UID: \"080f5dc7-c78f-4c7f-ab59-e7e05988b487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f154566-b8n47"
Nov 02 01:49:18.885409 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:18.885250    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080f5dc7-c78f-4c7f-ab59-e7e05988b487-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f154566-b8n47\" (UID: \"080f5dc7-c78f-4c7f-ab59-e7e05988b487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f154566-b8n47"
Nov 02 01:49:18.885409 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:18.885369    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080f5dc7-c78f-4c7f-ab59-e7e05988b487-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f154566-b8n47\" (UID: \"080f5dc7-c78f-4c7f-ab59-e7e05988b487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f154566-b8n47"
Nov 02 01:49:18.885908 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:18.885884    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080f5dc7-c78f-4c7f-ab59-e7e05988b487-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f154566-b8n47\" (UID: \"080f5dc7-c78f-4c7f-ab59-e7e05988b487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f154566-b8n47"
Nov 02 01:49:18.888029 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:18.888008    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080f5dc7-c78f-4c7f-ab59-e7e05988b487-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f154566-b8n47\" (UID: \"080f5dc7-c78f-4c7f-ab59-e7e05988b487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f154566-b8n47"
Nov 02 01:49:18.947365 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:18.947296    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkpb2\" (UniqueName: \"kubernetes.io/projected/080f5dc7-c78f-4c7f-ab59-e7e05988b487-kube-api-access-zkpb2\") pod \"maas-api-revoke-sub-9f154566-b8n47\" (UID: \"080f5dc7-c78f-4c7f-ab59-e7e05988b487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f154566-b8n47"
Nov 02 01:49:19.072236 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:19.072154    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f154566-b8n47"
Nov 02 01:49:19.455921 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:49:19.455885    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080f5dc7_c78f_4c7f_ab59_e7e05988b487.slice/crio-bd232058f6ccf85f219c220718dc9aa970875c254519289951f0c8d30d76507a WatchSource:0}: Error finding container bd232058f6ccf85f219c220718dc9aa970875c254519289951f0c8d30d76507a: Status 404 returned error can't find the container with id bd232058f6ccf85f219c220718dc9aa970875c254519289951f0c8d30d76507a
Nov 02 01:49:19.457268 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:19.457240    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f154566-b8n47"]
Nov 02 01:49:19.920328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:19.919509    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f154566-b8n47" event={"ID":"080f5dc7-c78f-4c7f-ab59-e7e05988b487","Type":"ContainerStarted","Data":"7759776f4a462e961d68eb80f6e7c1dc5731bf450e3ecdd3d093be4f414e7a3a"}
Nov 02 01:49:19.920328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:19.919585    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f154566-b8n47" event={"ID":"080f5dc7-c78f-4c7f-ab59-e7e05988b487","Type":"ContainerStarted","Data":"bd232058f6ccf85f219c220718dc9aa970875c254519289951f0c8d30d76507a"}
Nov 02 01:49:20.035741 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:20.035639    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f154566-b8n47" podStartSLOduration=2.035624025 podStartE2EDuration="2.035624025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:20.034452146 +0000 UTC m=+1424.938134260" watchObservedRunningTime="2026-11-02 01:49:20.035624025 +0000 UTC m=+1424.939306138"
Nov 02 01:49:20.932739 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:20.932702    5639 generic.go:356] "Generic (PLEG): container finished" podID="080f5dc7-c78f-4c7f-ab59-e7e05988b487" containerID="7759776f4a462e961d68eb80f6e7c1dc5731bf450e3ecdd3d093be4f414e7a3a" exitCode=0
Nov 02 01:49:20.933164 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:20.932764    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f154566-b8n47" event={"ID":"080f5dc7-c78f-4c7f-ab59-e7e05988b487","Type":"ContainerDied","Data":"7759776f4a462e961d68eb80f6e7c1dc5731bf450e3ecdd3d093be4f414e7a3a"}
Nov 02 01:49:22.103869 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:22.103843    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f154566-b8n47"
Nov 02 01:49:22.141694 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:22.141663    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/080f5dc7-c78f-4c7f-ab59-e7e05988b487-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080f5dc7-c78f-4c7f-ab59-e7e05988b487-maas-api-openshift-ca\") pod \"080f5dc7-c78f-4c7f-ab59-e7e05988b487\" (UID: \"080f5dc7-c78f-4c7f-ab59-e7e05988b487\") "
Nov 02 01:49:22.141877 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:22.141858    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/080f5dc7-c78f-4c7f-ab59-e7e05988b487-kube-api-access-zkpb2\" (UniqueName: \"kubernetes.io/projected/080f5dc7-c78f-4c7f-ab59-e7e05988b487-kube-api-access-zkpb2\") pod \"080f5dc7-c78f-4c7f-ab59-e7e05988b487\" (UID: \"080f5dc7-c78f-4c7f-ab59-e7e05988b487\") "
Nov 02 01:49:22.141972 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:22.141959    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/080f5dc7-c78f-4c7f-ab59-e7e05988b487-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080f5dc7-c78f-4c7f-ab59-e7e05988b487-maas-api-xks-ca\") pod \"080f5dc7-c78f-4c7f-ab59-e7e05988b487\" (UID: \"080f5dc7-c78f-4c7f-ab59-e7e05988b487\") "
Nov 02 01:49:22.142057 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:22.142030    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/080f5dc7-c78f-4c7f-ab59-e7e05988b487-maas-api-openshift-ca" pod "080f5dc7-c78f-4c7f-ab59-e7e05988b487" (UID: "080f5dc7-c78f-4c7f-ab59-e7e05988b487"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:22.142680 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:22.142659    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080f5dc7-c78f-4c7f-ab59-e7e05988b487-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:22.144624 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:22.144596    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080f5dc7-c78f-4c7f-ab59-e7e05988b487-kube-api-access-zkpb2" pod "080f5dc7-c78f-4c7f-ab59-e7e05988b487" (UID: "080f5dc7-c78f-4c7f-ab59-e7e05988b487"). InnerVolumeSpecName "kube-api-access-zkpb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:22.144624 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:22.144615    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/080f5dc7-c78f-4c7f-ab59-e7e05988b487-maas-api-xks-ca" pod "080f5dc7-c78f-4c7f-ab59-e7e05988b487" (UID: "080f5dc7-c78f-4c7f-ab59-e7e05988b487"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:22.244669 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:22.244586    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080f5dc7-c78f-4c7f-ab59-e7e05988b487-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:22.244669 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:22.244625    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkpb2\" (UniqueName: \"kubernetes.io/projected/080f5dc7-c78f-4c7f-ab59-e7e05988b487-kube-api-access-zkpb2\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:22.952038 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:22.952009    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f154566-b8n47"
Nov 02 01:49:22.952218 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:22.952001    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f154566-b8n47" event={"ID":"080f5dc7-c78f-4c7f-ab59-e7e05988b487","Type":"ContainerDied","Data":"bd232058f6ccf85f219c220718dc9aa970875c254519289951f0c8d30d76507a"}
Nov 02 01:49:22.952218 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:22.952167    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd232058f6ccf85f219c220718dc9aa970875c254519289951f0c8d30d76507a"
Nov 02 01:49:25.043954 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.043922    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"]
Nov 02 01:49:25.044650 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.044633    5639 cpu_manager.go:414] "containerMap: removing container" podUID="080f5dc7-c78f-4c7f-ab59-e7e05988b487" containerName="revoke-keys"
Nov 02 01:49:25.044712 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.044651    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="080f5dc7-c78f-4c7f-ab59-e7e05988b487" containerName="revoke-keys"
Nov 02 01:49:25.044782 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.044773    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="080f5dc7-c78f-4c7f-ab59-e7e05988b487" containerName="revoke-keys"
Nov 02 01:49:25.048525 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.048504    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.153401 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.153369    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"]
Nov 02 01:49:25.188843 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.188809    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.188843 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.188850    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.189118 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.189055    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.189118 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.189115    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ab0dca-9dde-45ae-8647-61a346c461f2-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.189238 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.189138    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84hgc\" (UniqueName: \"kubernetes.io/projected/e5ab0dca-9dde-45ae-8647-61a346c461f2-kube-api-access-84hgc\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.189238 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.189159    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.189238 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.189215    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.189426 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.189319    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.189426 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.189364    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.291038 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.290998    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.291236 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.291067    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ab0dca-9dde-45ae-8647-61a346c461f2-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.291236 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.291100    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84hgc\" (UniqueName: \"kubernetes.io/projected/e5ab0dca-9dde-45ae-8647-61a346c461f2-kube-api-access-84hgc\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.291236 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.291145    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.291236 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.291198    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.291236 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.291226    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.291539 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.291311    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.291539 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.291457    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.291539 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.291520    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.291704 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.291619    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.291704 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.291658    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.291704 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.291693    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.291970 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.291946    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ab0dca-9dde-45ae-8647-61a346c461f2-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.292087 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.292064    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.293835 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.293818    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.294089 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.294050    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.359371 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.359334    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84hgc\" (UniqueName: \"kubernetes.io/projected/e5ab0dca-9dde-45ae-8647-61a346c461f2-kube-api-access-84hgc\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.363584 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.363551    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.389180 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.389145    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"]
Nov 02 01:49:25.395335 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.395315    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.489306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.477797    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"]
Nov 02 01:49:25.494993 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.494966    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.495311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.495273    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.495518 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.495501    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jhcs\" (UniqueName: \"kubernetes.io/projected/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-kube-api-access-5jhcs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.495617 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.495597    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.495750 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.495722    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.495819 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.495772    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.495819 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.495802    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.495929 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.495830    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.495929 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.495867    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.597443 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.597411    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.597620 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.597449    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.597620 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.597470    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.597620 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.597496    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.597872 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.597627    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.597872 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.597689    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.597872 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.597824    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jhcs\" (UniqueName: \"kubernetes.io/projected/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-kube-api-access-5jhcs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.598034 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.597910    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.598034 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.597926    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.598034 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.597948    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.598034 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.598000    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.598225 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.598049    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.598225 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.598072    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.598806 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.598783    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.600139 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.600121    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.600336 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.600271    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.659809 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.659777    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.659956 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.659876    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jhcs\" (UniqueName: \"kubernetes.io/projected/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-kube-api-access-5jhcs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:25.662781 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.662763    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:25.710793 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:25.710755    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:26.146312 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:26.146148    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"]
Nov 02 01:49:26.150320 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:26.148343    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:49:26.150320 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:26.148454    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:49:26.150320 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:26.148505    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:49:26.199917 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:26.199883    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"]
Nov 02 01:49:26.203093 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:26.203051    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:49:26.203222 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:26.203136    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:49:26.203222 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:26.203182    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:49:27.021235 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:27.020832    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc" event={"ID":"e5ab0dca-9dde-45ae-8647-61a346c461f2","Type":"ContainerStarted","Data":"746680d693e3d8728af7b852f710d2a687337ab69215189cfd91fe21c0b1c74e"}
Nov 02 01:49:27.021235 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:27.020911    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc" event={"ID":"e5ab0dca-9dde-45ae-8647-61a346c461f2","Type":"ContainerStarted","Data":"aebc71c0afb162a06de2dc24da1586d98bcf2fa7625c3a86c0ffea03cb25fd7d"}
Nov 02 01:49:27.022967 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:27.022938    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4" event={"ID":"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb","Type":"ContainerStarted","Data":"8f2444dbcb8a20f1a1811ccec632e5faf22156fd547d64079fd789e259aeda5c"}
Nov 02 01:49:27.023104 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:27.022987    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4" event={"ID":"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb","Type":"ContainerStarted","Data":"9567639b1f97913f7991a73d9becd5e644a511a15da705182cf7c80c0d7fad94"}
Nov 02 01:49:27.148015 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:27.147957    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc" podStartSLOduration=3.1479406 podStartE2EDuration="3.1479406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:27.147641461 +0000 UTC m=+1432.051323574" watchObservedRunningTime="2026-11-02 01:49:27.1479406 +0000 UTC m=+1432.051622714"
Nov 02 01:49:27.310370 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:27.306032    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4" podStartSLOduration=2.306010678 podStartE2EDuration="2.306010678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:27.285671181 +0000 UTC m=+1432.189353295" watchObservedRunningTime="2026-11-02 01:49:27.306010678 +0000 UTC m=+1432.209692792"
Nov 02 01:49:27.439224 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:49:27.439190    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:49:27.678232 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:27.675658    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:27.683306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:27.679395    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 01:49:27.683306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:27.679454    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc" podUID="e5ab0dca-9dde-45ae-8647-61a346c461f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 01:49:27.711731 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:27.711598    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:27.712162 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:27.712138    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 02 01:49:27.712267 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:27.712185    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4" podUID="2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 02 01:49:28.669307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:28.667795    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 01:49:28.669307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:28.667863    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc" podUID="e5ab0dca-9dde-45ae-8647-61a346c461f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 01:49:28.711801 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:28.711758    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 02 01:49:28.711979 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:28.711828    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4" podUID="2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 02 01:49:29.680304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:29.672681    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 01:49:29.680304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:29.672745    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc" podUID="e5ab0dca-9dde-45ae-8647-61a346c461f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 01:49:29.712142 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:29.712058    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 02 01:49:29.712142 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:29.712127    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4" podUID="2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 02 01:49:30.673322 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:30.673261    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 01:49:30.673644 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:30.673615    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc" podUID="e5ab0dca-9dde-45ae-8647-61a346c461f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 01:49:30.712734 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:30.712689    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 02 01:49:30.713140 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:30.712757    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4" podUID="2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 02 01:49:31.670315 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:31.663723    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 01:49:31.670315 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:31.663791    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc" podUID="e5ab0dca-9dde-45ae-8647-61a346c461f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 01:49:31.711876 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:31.711727    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 02 01:49:31.711876 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:31.711798    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4" podUID="2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 02 01:49:32.664165 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:32.664126    5639 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 01:49:32.664640 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:32.664185    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc" podUID="e5ab0dca-9dde-45ae-8647-61a346c461f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 01:49:32.711473 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:32.711431    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 02 01:49:32.711656 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:32.711501    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4" podUID="2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 02 01:49:33.716196 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:33.716164    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:33.716794 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:33.716775    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:33.717611 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:33.717592    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:49:33.723380 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:33.723339    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:33.724046 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:33.724014    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:33.724907 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:33.724874    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:49:35.623089 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.623057    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ffb490f-8tplc"]
Nov 02 01:49:35.629219 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.629199    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffb490f-8tplc"
Nov 02 01:49:35.683677 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.683648    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ffb490f-8tplc"]
Nov 02 01:49:35.683859 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.683753    5639 scope.go:122] "RemoveContainer" containerID="899cd3e1f5d00a8e120afd2e4130d074f693f7fce584a84555d68f7b7e973bf7"
Nov 02 01:49:35.699557 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.699527    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093e8792-ba76-40e9-aad5-90b314440ca8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ffb490f-8tplc\" (UID: \"093e8792-ba76-40e9-aad5-90b314440ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffb490f-8tplc"
Nov 02 01:49:35.699803 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.699775    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vskn9\" (UniqueName: \"kubernetes.io/projected/093e8792-ba76-40e9-aad5-90b314440ca8-kube-api-access-vskn9\") pod \"maas-api-revoke-sub-4ffb490f-8tplc\" (UID: \"093e8792-ba76-40e9-aad5-90b314440ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffb490f-8tplc"
Nov 02 01:49:35.699935 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.699920    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093e8792-ba76-40e9-aad5-90b314440ca8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ffb490f-8tplc\" (UID: \"093e8792-ba76-40e9-aad5-90b314440ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffb490f-8tplc"
Nov 02 01:49:35.700258 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.700240    5639 scope.go:122] "RemoveContainer" containerID="9e5cbdb374459051e3684c5a21231ef03076a128c18a0abe830884144d8ba601"
Nov 02 01:49:35.717351 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.717327    5639 scope.go:122] "RemoveContainer" containerID="a2f6b4985511549b042efd6849126ae556e3ea33d98e8adea2d601b53d1b63c1"
Nov 02 01:49:35.727236 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.727210    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:49:35.727449 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.727212    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:49:35.727449 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.727217    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:49:35.727449 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.727258    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:49:35.728345 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.728330    5639 scope.go:122] "RemoveContainer" containerID="b9e79d4813b5d86da4971039b53ec183ce880b4e55a1a5138d0403c1ae5cff86"
Nov 02 01:49:35.738110 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.738090    5639 scope.go:122] "RemoveContainer" containerID="3fc8f30239fe3d46d9377fddfb41b3380d1b00a5b5105254c1820c1c3fd9e2f9"
Nov 02 01:49:35.749096 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.749074    5639 scope.go:122] "RemoveContainer" containerID="3659dba05dcc908d64e2a188b9db3fa1b57bfcc464f41ae1ba06e64132555280"
Nov 02 01:49:35.759644 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.759619    5639 scope.go:122] "RemoveContainer" containerID="1e5eced2de1d1e8522d28a6bec2a74dc4f658300df956fe45054776b8eeb3f00"
Nov 02 01:49:35.769493 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.769474    5639 scope.go:122] "RemoveContainer" containerID="9f3f53b5da29d89a03b7a85b299aeb732627e9d3e36727b3bc2aae20db4ad467"
Nov 02 01:49:35.778956 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.778935    5639 scope.go:122] "RemoveContainer" containerID="15d414964e543a06c555f6b5c02938a0cb8438147ac38cf41e8062fc11bc1d18"
Nov 02 01:49:35.788509 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.788482    5639 scope.go:122] "RemoveContainer" containerID="89cfd726dfcdb32b8e7c79ebbec5ee77cfc09951935425962bb67b80cbb8aff1"
Nov 02 01:49:35.797881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.797862    5639 scope.go:122] "RemoveContainer" containerID="32b11cf56ba7cd375c46a5733d83f33c48a1c9a778358c34006c5716cdba9328"
Nov 02 01:49:35.802169 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.802143    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093e8792-ba76-40e9-aad5-90b314440ca8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ffb490f-8tplc\" (UID: \"093e8792-ba76-40e9-aad5-90b314440ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffb490f-8tplc"
Nov 02 01:49:35.802438 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.802414    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vskn9\" (UniqueName: \"kubernetes.io/projected/093e8792-ba76-40e9-aad5-90b314440ca8-kube-api-access-vskn9\") pod \"maas-api-revoke-sub-4ffb490f-8tplc\" (UID: \"093e8792-ba76-40e9-aad5-90b314440ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffb490f-8tplc"
Nov 02 01:49:35.802613 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.802593    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093e8792-ba76-40e9-aad5-90b314440ca8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ffb490f-8tplc\" (UID: \"093e8792-ba76-40e9-aad5-90b314440ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffb490f-8tplc"
Nov 02 01:49:35.803426 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.803405    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093e8792-ba76-40e9-aad5-90b314440ca8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ffb490f-8tplc\" (UID: \"093e8792-ba76-40e9-aad5-90b314440ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffb490f-8tplc"
Nov 02 01:49:35.805121 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.805099    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093e8792-ba76-40e9-aad5-90b314440ca8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ffb490f-8tplc\" (UID: \"093e8792-ba76-40e9-aad5-90b314440ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffb490f-8tplc"
Nov 02 01:49:35.808917 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.808898    5639 scope.go:122] "RemoveContainer" containerID="b48ae12da37af2f5f102152c5339fb41edd7e4a81cec92b276f919850a57092a"
Nov 02 01:49:35.830869 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.830844    5639 scope.go:122] "RemoveContainer" containerID="bebd2e8d892d44bc60243e2c27409d3171bb3167c302e3549552a65c41166a6d"
Nov 02 01:49:35.863807 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.863785    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vskn9\" (UniqueName: \"kubernetes.io/projected/093e8792-ba76-40e9-aad5-90b314440ca8-kube-api-access-vskn9\") pod \"maas-api-revoke-sub-4ffb490f-8tplc\" (UID: \"093e8792-ba76-40e9-aad5-90b314440ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffb490f-8tplc"
Nov 02 01:49:35.943860 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:35.943838    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffb490f-8tplc"
Nov 02 01:49:36.528364 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:49:36.528329    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093e8792_ba76_40e9_aad5_90b314440ca8.slice/crio-51e8feab6346b8fdffd7f899312785ee0eb91fde930cb99613b68dc5e3472c85 WatchSource:0}: Error finding container 51e8feab6346b8fdffd7f899312785ee0eb91fde930cb99613b68dc5e3472c85: Status 404 returned error can't find the container with id 51e8feab6346b8fdffd7f899312785ee0eb91fde930cb99613b68dc5e3472c85
Nov 02 01:49:36.529052 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:36.529026    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ffb490f-8tplc"]
Nov 02 01:49:37.256096 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:37.256062    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffb490f-8tplc" event={"ID":"093e8792-ba76-40e9-aad5-90b314440ca8","Type":"ContainerStarted","Data":"4635aa259a44fee93719d3edbe64ac42838f164d5d505073a29850fdc8f7ce1c"}
Nov 02 01:49:37.256537 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:37.256124    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffb490f-8tplc" event={"ID":"093e8792-ba76-40e9-aad5-90b314440ca8","Type":"ContainerStarted","Data":"51e8feab6346b8fdffd7f899312785ee0eb91fde930cb99613b68dc5e3472c85"}
Nov 02 01:49:37.374141 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:37.374084    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffb490f-8tplc" podStartSLOduration=2.374068516 podStartE2EDuration="2.374068516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:37.371777134 +0000 UTC m=+1442.275459276" watchObservedRunningTime="2026-11-02 01:49:37.374068516 +0000 UTC m=+1442.277750629"
Nov 02 01:49:38.336559 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:38.336490    5639 generic.go:356] "Generic (PLEG): container finished" podID="093e8792-ba76-40e9-aad5-90b314440ca8" containerID="4635aa259a44fee93719d3edbe64ac42838f164d5d505073a29850fdc8f7ce1c" exitCode=0
Nov 02 01:49:38.337039 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:38.336634    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffb490f-8tplc" event={"ID":"093e8792-ba76-40e9-aad5-90b314440ca8","Type":"ContainerDied","Data":"4635aa259a44fee93719d3edbe64ac42838f164d5d505073a29850fdc8f7ce1c"}
Nov 02 01:49:40.676847 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:40.676810    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffb490f-8tplc"
Nov 02 01:49:40.828362 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:40.822875    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/093e8792-ba76-40e9-aad5-90b314440ca8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093e8792-ba76-40e9-aad5-90b314440ca8-maas-api-openshift-ca\") pod \"093e8792-ba76-40e9-aad5-90b314440ca8\" (UID: \"093e8792-ba76-40e9-aad5-90b314440ca8\") "
Nov 02 01:49:40.828362 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:40.822951    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/093e8792-ba76-40e9-aad5-90b314440ca8-kube-api-access-vskn9\" (UniqueName: \"kubernetes.io/projected/093e8792-ba76-40e9-aad5-90b314440ca8-kube-api-access-vskn9\") pod \"093e8792-ba76-40e9-aad5-90b314440ca8\" (UID: \"093e8792-ba76-40e9-aad5-90b314440ca8\") "
Nov 02 01:49:40.828362 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:40.823172    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/093e8792-ba76-40e9-aad5-90b314440ca8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093e8792-ba76-40e9-aad5-90b314440ca8-maas-api-xks-ca\") pod \"093e8792-ba76-40e9-aad5-90b314440ca8\" (UID: \"093e8792-ba76-40e9-aad5-90b314440ca8\") "
Nov 02 01:49:40.828362 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:40.825185    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/093e8792-ba76-40e9-aad5-90b314440ca8-maas-api-openshift-ca" pod "093e8792-ba76-40e9-aad5-90b314440ca8" (UID: "093e8792-ba76-40e9-aad5-90b314440ca8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:40.841213 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:40.835791    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093e8792-ba76-40e9-aad5-90b314440ca8-kube-api-access-vskn9" pod "093e8792-ba76-40e9-aad5-90b314440ca8" (UID: "093e8792-ba76-40e9-aad5-90b314440ca8"). InnerVolumeSpecName "kube-api-access-vskn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:40.841213 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:40.836377    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/093e8792-ba76-40e9-aad5-90b314440ca8-maas-api-xks-ca" pod "093e8792-ba76-40e9-aad5-90b314440ca8" (UID: "093e8792-ba76-40e9-aad5-90b314440ca8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:40.927127 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:40.927046    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093e8792-ba76-40e9-aad5-90b314440ca8-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:40.927127 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:40.927087    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093e8792-ba76-40e9-aad5-90b314440ca8-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:40.927127 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:40.927103    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vskn9\" (UniqueName: \"kubernetes.io/projected/093e8792-ba76-40e9-aad5-90b314440ca8-kube-api-access-vskn9\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:41.636372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:41.636006    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffb490f-8tplc" event={"ID":"093e8792-ba76-40e9-aad5-90b314440ca8","Type":"ContainerDied","Data":"51e8feab6346b8fdffd7f899312785ee0eb91fde930cb99613b68dc5e3472c85"}
Nov 02 01:49:41.636372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:41.636073    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51e8feab6346b8fdffd7f899312785ee0eb91fde930cb99613b68dc5e3472c85"
Nov 02 01:49:41.636372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:41.636142    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffb490f-8tplc"
Nov 02 01:49:46.417186 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:46.413904    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2804cba7-8hgcq"]
Nov 02 01:49:46.417186 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:46.415134    5639 cpu_manager.go:414] "containerMap: removing container" podUID="093e8792-ba76-40e9-aad5-90b314440ca8" containerName="revoke-keys"
Nov 02 01:49:46.417186 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:46.415157    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093e8792-ba76-40e9-aad5-90b314440ca8" containerName="revoke-keys"
Nov 02 01:49:46.417186 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:46.415355    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="093e8792-ba76-40e9-aad5-90b314440ca8" containerName="revoke-keys"
Nov 02 01:49:46.429350 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:46.429321    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2804cba7-8hgcq"
Nov 02 01:49:46.465579 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:46.465536    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2804cba7-8hgcq\" (UID: \"bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2804cba7-8hgcq"
Nov 02 01:49:46.480168 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:46.465892    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2804cba7-8hgcq\" (UID: \"bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2804cba7-8hgcq"
Nov 02 01:49:46.480547 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:46.480515    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxxck\" (UniqueName: \"kubernetes.io/projected/bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a-kube-api-access-cxxck\") pod \"maas-api-revoke-sub-2804cba7-8hgcq\" (UID: \"bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2804cba7-8hgcq"
Nov 02 01:49:46.497652 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:46.476373    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2804cba7-8hgcq"]
Nov 02 01:49:46.540344 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:46.540254    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:49:46.540557 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:46.540294    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:49:46.540557 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:46.540316    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:49:46.540557 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:46.540359    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:49:46.599361 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:46.599322    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2804cba7-8hgcq\" (UID: \"bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2804cba7-8hgcq"
Nov 02 01:49:46.599599 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:46.599416    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2804cba7-8hgcq\" (UID: \"bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2804cba7-8hgcq"
Nov 02 01:49:46.599599 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:46.599470    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxxck\" (UniqueName: \"kubernetes.io/projected/bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a-kube-api-access-cxxck\") pod \"maas-api-revoke-sub-2804cba7-8hgcq\" (UID: \"bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2804cba7-8hgcq"
Nov 02 01:49:46.600014 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:46.599991    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2804cba7-8hgcq\" (UID: \"bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2804cba7-8hgcq"
Nov 02 01:49:46.602098 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:46.602068    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2804cba7-8hgcq\" (UID: \"bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2804cba7-8hgcq"
Nov 02 01:49:46.660326 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:46.660253    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxxck\" (UniqueName: \"kubernetes.io/projected/bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a-kube-api-access-cxxck\") pod \"maas-api-revoke-sub-2804cba7-8hgcq\" (UID: \"bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2804cba7-8hgcq"
Nov 02 01:49:46.804977 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:46.804873    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2804cba7-8hgcq"
Nov 02 01:49:47.449309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:47.449060    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-59d70a90-d54879985-jv9l7"]
Nov 02 01:49:47.471301 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:47.461492    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-59d70a90-d54879985-jv9l7"
Nov 02 01:49:47.597727 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:47.597681    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-59d70a90-dockercfg-h2xhh\""
Nov 02 01:49:47.598763 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:47.598640    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-59d70a90\""
Nov 02 01:49:47.629812 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:47.629776    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060e39cf-da59-4ca7-9e12-50b88e7e4cad-plugins-config-volume\") pod \"payload-processing-e2e-ait-59d70a90-d54879985-jv9l7\" (UID: \"060e39cf-da59-4ca7-9e12-50b88e7e4cad\") " pod="openshift-ingress/payload-processing-e2e-ait-59d70a90-d54879985-jv9l7"
Nov 02 01:49:47.629973 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:47.629851    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmbxm\" (UniqueName: \"kubernetes.io/projected/060e39cf-da59-4ca7-9e12-50b88e7e4cad-kube-api-access-dmbxm\") pod \"payload-processing-e2e-ait-59d70a90-d54879985-jv9l7\" (UID: \"060e39cf-da59-4ca7-9e12-50b88e7e4cad\") " pod="openshift-ingress/payload-processing-e2e-ait-59d70a90-d54879985-jv9l7"
Nov 02 01:49:47.631113 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:47.631083    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-59d70a90-d54879985-jv9l7"]
Nov 02 01:49:47.641124 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:47.641097    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2804cba7-8hgcq"]
Nov 02 01:49:47.647305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:47.647254    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd"]
Nov 02 01:49:47.656517 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:47.656494    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd"]
Nov 02 01:49:47.656648 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:47.656603    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd"
Nov 02 01:49:47.774310 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:47.773652    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nww7\" (UniqueName: \"kubernetes.io/projected/b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73-kube-api-access-8nww7\") pod \"payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd\" (UID: \"b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd"
Nov 02 01:49:47.774310 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:47.773777    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd\" (UID: \"b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd"
Nov 02 01:49:47.803396 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:47.786607    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060e39cf-da59-4ca7-9e12-50b88e7e4cad-plugins-config-volume\") pod \"payload-processing-e2e-ait-59d70a90-d54879985-jv9l7\" (UID: \"060e39cf-da59-4ca7-9e12-50b88e7e4cad\") " pod="openshift-ingress/payload-processing-e2e-ait-59d70a90-d54879985-jv9l7"
Nov 02 01:49:47.803396 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:47.786703    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmbxm\" (UniqueName: \"kubernetes.io/projected/060e39cf-da59-4ca7-9e12-50b88e7e4cad-kube-api-access-dmbxm\") pod \"payload-processing-e2e-ait-59d70a90-d54879985-jv9l7\" (UID: \"060e39cf-da59-4ca7-9e12-50b88e7e4cad\") " pod="openshift-ingress/payload-processing-e2e-ait-59d70a90-d54879985-jv9l7"
Nov 02 01:49:47.803396 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:47.792880    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060e39cf-da59-4ca7-9e12-50b88e7e4cad-plugins-config-volume\") pod \"payload-processing-e2e-ait-59d70a90-d54879985-jv9l7\" (UID: \"060e39cf-da59-4ca7-9e12-50b88e7e4cad\") " pod="openshift-ingress/payload-processing-e2e-ait-59d70a90-d54879985-jv9l7"
Nov 02 01:49:47.937363 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:47.926865    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd\" (UID: \"b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd"
Nov 02 01:49:47.937363 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:47.927325    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nww7\" (UniqueName: \"kubernetes.io/projected/b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73-kube-api-access-8nww7\") pod \"payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd\" (UID: \"b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd"
Nov 02 01:49:47.968368 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:47.967152    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd\" (UID: \"b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd"
Nov 02 01:49:48.007105 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:48.007080    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmbxm\" (UniqueName: \"kubernetes.io/projected/060e39cf-da59-4ca7-9e12-50b88e7e4cad-kube-api-access-dmbxm\") pod \"payload-processing-e2e-ait-59d70a90-d54879985-jv9l7\" (UID: \"060e39cf-da59-4ca7-9e12-50b88e7e4cad\") " pod="openshift-ingress/payload-processing-e2e-ait-59d70a90-d54879985-jv9l7"
Nov 02 01:49:48.008911 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:48.007703    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2804cba7-8hgcq" event={"ID":"bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a","Type":"ContainerStarted","Data":"1b05e7d2c42e2901d4d0e8f2bbaf4f786e22e63d4174855b1d0e47957344dda1"}
Nov 02 01:49:48.011967 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:48.011940    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nww7\" (UniqueName: \"kubernetes.io/projected/b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73-kube-api-access-8nww7\") pod \"payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd\" (UID: \"b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd"
Nov 02 01:49:48.246554 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:48.246079    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-59d70a90-d54879985-jv9l7"
Nov 02 01:49:48.286268 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:48.286241    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd"
Nov 02 01:49:48.688306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:48.685684    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-59d70a90-d54879985-jv9l7"]
Nov 02 01:49:48.712650 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:48.712609    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd"]
Nov 02 01:49:48.727043 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:49:48.727015    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2ede435_f1d8_4c62_b0df_d7a2c9e7cb73.slice/crio-64fa45da471fbab24b380e965e9a493b612bb79801bd34df36fc02a4d57483e6 WatchSource:0}: Error finding container 64fa45da471fbab24b380e965e9a493b612bb79801bd34df36fc02a4d57483e6: Status 404 returned error can't find the container with id 64fa45da471fbab24b380e965e9a493b612bb79801bd34df36fc02a4d57483e6
Nov 02 01:49:49.017527 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:49.017432    5639 generic.go:356] "Generic (PLEG): container finished" podID="bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a" containerID="5a60374ed76a0dc1ce621130abf9562507f94e2d2c12312830b1a5be6ee97461" exitCode=0
Nov 02 01:49:49.017698 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:49.017519    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2804cba7-8hgcq" event={"ID":"bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a","Type":"ContainerDied","Data":"5a60374ed76a0dc1ce621130abf9562507f94e2d2c12312830b1a5be6ee97461"}
Nov 02 01:49:49.019582 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:49.019552    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-59d70a90-d54879985-jv9l7" event={"ID":"060e39cf-da59-4ca7-9e12-50b88e7e4cad","Type":"ContainerStarted","Data":"c22524ed377e3dc969e68ef431eb38e511f4ecc12248fb3a5966127a0797b0b1"}
Nov 02 01:49:49.019755 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:49.019605    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-59d70a90-d54879985-jv9l7" event={"ID":"060e39cf-da59-4ca7-9e12-50b88e7e4cad","Type":"ContainerStarted","Data":"9abc11a79afe297c5da52d7d851cd705a77eeacba4de0110067f367ee4f1524e"}
Nov 02 01:49:49.019755 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:49.019638    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-59d70a90-d54879985-jv9l7"
Nov 02 01:49:49.021629 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:49.021603    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd" event={"ID":"b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73","Type":"ContainerStarted","Data":"1ab7fc9ea3c81e8ef07d5c8aa70fe273ab739151cff833d3421298c689e3c454"}
Nov 02 01:49:49.021759 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:49.021652    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd" event={"ID":"b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73","Type":"ContainerStarted","Data":"64fa45da471fbab24b380e965e9a493b612bb79801bd34df36fc02a4d57483e6"}
Nov 02 01:49:49.021759 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:49.021678    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd"
Nov 02 01:49:49.428047 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:49.427965    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-59d70a90-d54879985-jv9l7" podStartSLOduration=2.427938427 podStartE2EDuration="2.427938427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:49.412984751 +0000 UTC m=+1454.316666872" watchObservedRunningTime="2026-11-02 01:49:49.427938427 +0000 UTC m=+1454.331620542"
Nov 02 01:49:49.432337 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:49.429241    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd" podStartSLOduration=2.429224422 podStartE2EDuration="2.429224422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:49.296253432 +0000 UTC m=+1454.199935547" watchObservedRunningTime="2026-11-02 01:49:49.429224422 +0000 UTC m=+1454.332906538"
Nov 02 01:49:50.192778 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:50.192754    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2804cba7-8hgcq"
Nov 02 01:49:50.326484 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:50.326387    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a-maas-api-xks-ca\") pod \"bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a\" (UID: \"bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a\") "
Nov 02 01:49:50.326761 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:50.326741    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a-maas-api-openshift-ca\") pod \"bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a\" (UID: \"bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a\") "
Nov 02 01:49:50.326849 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:50.326797    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a-kube-api-access-cxxck\" (UniqueName: \"kubernetes.io/projected/bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a-kube-api-access-cxxck\") pod \"bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a\" (UID: \"bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a\") "
Nov 02 01:49:50.329308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:50.327685    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a-maas-api-openshift-ca" pod "bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a" (UID: "bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:50.329308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:50.328015    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:50.334521 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:50.334476    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a-maas-api-xks-ca" pod "bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a" (UID: "bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:50.344519 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:50.344476    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a-kube-api-access-cxxck" pod "bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a" (UID: "bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a"). InnerVolumeSpecName "kube-api-access-cxxck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:50.429821 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:50.429776    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxxck\" (UniqueName: \"kubernetes.io/projected/bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a-kube-api-access-cxxck\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:50.429821 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:50.429817    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:51.039719 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:51.039688    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2804cba7-8hgcq"
Nov 02 01:49:51.039935 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:51.039716    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2804cba7-8hgcq" event={"ID":"bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a","Type":"ContainerDied","Data":"1b05e7d2c42e2901d4d0e8f2bbaf4f786e22e63d4174855b1d0e47957344dda1"}
Nov 02 01:49:51.039935 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:51.039786    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b05e7d2c42e2901d4d0e8f2bbaf4f786e22e63d4174855b1d0e47957344dda1"
Nov 02 01:49:52.439306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:52.438689    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr"]
Nov 02 01:49:52.494572 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:52.439946    5639 cpu_manager.go:414] "containerMap: removing container" podUID="bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a" containerName="revoke-keys"
Nov 02 01:49:52.497524 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:52.497471    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a" containerName="revoke-keys"
Nov 02 01:49:52.498117 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:52.498095    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a" containerName="revoke-keys"
Nov 02 01:49:52.504040 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:52.504019    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr"]
Nov 02 01:49:52.504309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:52.504260    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr"
Nov 02 01:49:52.572503 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:52.570582    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp"]
Nov 02 01:49:52.584547 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:52.581962    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp"]
Nov 02 01:49:52.584547 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:52.582085    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp"
Nov 02 01:49:52.613965 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:52.612749    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2821b375\""
Nov 02 01:49:52.613965 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:52.612914    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2821b375-dockercfg-txvvx\""
Nov 02 01:49:52.676307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:52.668115    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79481b1b-92f4-4dc0-8ac1-b03e509606c3-plugins-config-volume\") pod \"payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp\" (UID: \"79481b1b-92f4-4dc0-8ac1-b03e509606c3\") " pod="openshift-ingress/payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp"
Nov 02 01:49:52.676307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:52.668333    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjfvl\" (UniqueName: \"kubernetes.io/projected/405c8e1f-2f4b-4624-ac6c-5f607e4bb81d-kube-api-access-pjfvl\") pod \"payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr\" (UID: \"405c8e1f-2f4b-4624-ac6c-5f607e4bb81d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr"
Nov 02 01:49:52.676307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:52.668392    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405c8e1f-2f4b-4624-ac6c-5f607e4bb81d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr\" (UID: \"405c8e1f-2f4b-4624-ac6c-5f607e4bb81d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr"
Nov 02 01:49:52.676307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:52.668492    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlrc7\" (UniqueName: \"kubernetes.io/projected/79481b1b-92f4-4dc0-8ac1-b03e509606c3-kube-api-access-tlrc7\") pod \"payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp\" (UID: \"79481b1b-92f4-4dc0-8ac1-b03e509606c3\") " pod="openshift-ingress/payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp"
Nov 02 01:49:52.776378 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:52.773753    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79481b1b-92f4-4dc0-8ac1-b03e509606c3-plugins-config-volume\") pod \"payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp\" (UID: \"79481b1b-92f4-4dc0-8ac1-b03e509606c3\") " pod="openshift-ingress/payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp"
Nov 02 01:49:52.776378 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:52.773848    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjfvl\" (UniqueName: \"kubernetes.io/projected/405c8e1f-2f4b-4624-ac6c-5f607e4bb81d-kube-api-access-pjfvl\") pod \"payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr\" (UID: \"405c8e1f-2f4b-4624-ac6c-5f607e4bb81d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr"
Nov 02 01:49:52.776378 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:52.773899    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405c8e1f-2f4b-4624-ac6c-5f607e4bb81d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr\" (UID: \"405c8e1f-2f4b-4624-ac6c-5f607e4bb81d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr"
Nov 02 01:49:52.776378 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:52.773996    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlrc7\" (UniqueName: \"kubernetes.io/projected/79481b1b-92f4-4dc0-8ac1-b03e509606c3-kube-api-access-tlrc7\") pod \"payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp\" (UID: \"79481b1b-92f4-4dc0-8ac1-b03e509606c3\") " pod="openshift-ingress/payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp"
Nov 02 01:49:52.776378 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:52.775112    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79481b1b-92f4-4dc0-8ac1-b03e509606c3-plugins-config-volume\") pod \"payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp\" (UID: \"79481b1b-92f4-4dc0-8ac1-b03e509606c3\") " pod="openshift-ingress/payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp"
Nov 02 01:49:52.776378 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:52.776070    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405c8e1f-2f4b-4624-ac6c-5f607e4bb81d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr\" (UID: \"405c8e1f-2f4b-4624-ac6c-5f607e4bb81d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr"
Nov 02 01:49:52.884339 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:52.880730    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjfvl\" (UniqueName: \"kubernetes.io/projected/405c8e1f-2f4b-4624-ac6c-5f607e4bb81d-kube-api-access-pjfvl\") pod \"payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr\" (UID: \"405c8e1f-2f4b-4624-ac6c-5f607e4bb81d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr"
Nov 02 01:49:52.920312 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:52.915452    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlrc7\" (UniqueName: \"kubernetes.io/projected/79481b1b-92f4-4dc0-8ac1-b03e509606c3-kube-api-access-tlrc7\") pod \"payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp\" (UID: \"79481b1b-92f4-4dc0-8ac1-b03e509606c3\") " pod="openshift-ingress/payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp"
Nov 02 01:49:52.942431 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:52.925560    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp"
Nov 02 01:49:53.140459 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:53.140426    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr"
Nov 02 01:49:53.388538 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:53.388512    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp"]
Nov 02 01:49:53.535937 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:49:53.535904    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405c8e1f_2f4b_4624_ac6c_5f607e4bb81d.slice/crio-a0e2088bec68963f2e08fb82be8ff0c8408db90a080a51967efcff7156d2cfa5 WatchSource:0}: Error finding container a0e2088bec68963f2e08fb82be8ff0c8408db90a080a51967efcff7156d2cfa5: Status 404 returned error can't find the container with id a0e2088bec68963f2e08fb82be8ff0c8408db90a080a51967efcff7156d2cfa5
Nov 02 01:49:53.537804 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:53.537781    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr"]
Nov 02 01:49:54.100777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:54.100736    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp" event={"ID":"79481b1b-92f4-4dc0-8ac1-b03e509606c3","Type":"ContainerStarted","Data":"721675fa48a2f22b0432bad608db84fbbb1cd85c6e34ac67fa25b2284b5c649c"}
Nov 02 01:49:54.100991 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:54.100804    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp" event={"ID":"79481b1b-92f4-4dc0-8ac1-b03e509606c3","Type":"ContainerStarted","Data":"9c90434981661a4d9b9f395e0299ee6f798ac625ded1c90ac6b9b7738d71876f"}
Nov 02 01:49:54.100991 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:54.100836    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp"
Nov 02 01:49:54.102583 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:54.102556    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr" event={"ID":"405c8e1f-2f4b-4624-ac6c-5f607e4bb81d","Type":"ContainerStarted","Data":"6f5bab36ea51048cec6ac63c4395edea8d7de93638234c03d77d93457eb0a134"}
Nov 02 01:49:54.102724 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:54.102613    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr"
Nov 02 01:49:54.102724 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:54.102638    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr" event={"ID":"405c8e1f-2f4b-4624-ac6c-5f607e4bb81d","Type":"ContainerStarted","Data":"a0e2088bec68963f2e08fb82be8ff0c8408db90a080a51967efcff7156d2cfa5"}
Nov 02 01:49:54.341596 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:54.341536    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp" podStartSLOduration=2.341516418 podStartE2EDuration="2.341516418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:54.221488432 +0000 UTC m=+1459.125170547" watchObservedRunningTime="2026-11-02 01:49:54.341516418 +0000 UTC m=+1459.245198533"
Nov 02 01:49:54.348266 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:54.347547    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr" podStartSLOduration=2.34752521 podStartE2EDuration="2.34752521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:54.340738724 +0000 UTC m=+1459.244420838" watchObservedRunningTime="2026-11-02 01:49:54.34752521 +0000 UTC m=+1459.251207328"
Nov 02 01:49:57.547118 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:49:57.547077    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:49:57.590315 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:57.578772    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv"]
Nov 02 01:49:57.619055 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:57.619012    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv"]
Nov 02 01:49:57.619478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:57.619440    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv"
Nov 02 01:49:57.653207 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:57.652788    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r85r\" (UniqueName: \"kubernetes.io/projected/ef2ff06b-7835-4075-9346-7dc5ced29cf4-kube-api-access-2r85r\") pod \"payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv\" (UID: \"ef2ff06b-7835-4075-9346-7dc5ced29cf4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv"
Nov 02 01:49:57.653207 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:57.652851    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2ff06b-7835-4075-9346-7dc5ced29cf4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv\" (UID: \"ef2ff06b-7835-4075-9346-7dc5ced29cf4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv"
Nov 02 01:49:57.714457 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:57.714390    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-fd723454\""
Nov 02 01:49:57.723216 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:57.722945    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2"]
Nov 02 01:49:57.727220 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:57.727172    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-fd723454-dockercfg-5xzsc\""
Nov 02 01:49:57.760092 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:57.754831    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r85r\" (UniqueName: \"kubernetes.io/projected/ef2ff06b-7835-4075-9346-7dc5ced29cf4-kube-api-access-2r85r\") pod \"payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv\" (UID: \"ef2ff06b-7835-4075-9346-7dc5ced29cf4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv"
Nov 02 01:49:57.760092 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:57.754914    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2ff06b-7835-4075-9346-7dc5ced29cf4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv\" (UID: \"ef2ff06b-7835-4075-9346-7dc5ced29cf4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv"
Nov 02 01:49:57.760092 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:57.756553    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2ff06b-7835-4075-9346-7dc5ced29cf4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv\" (UID: \"ef2ff06b-7835-4075-9346-7dc5ced29cf4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv"
Nov 02 01:49:57.789416 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:57.789241    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2"]
Nov 02 01:49:57.789416 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:57.789403    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2"
Nov 02 01:49:57.821396 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:57.821367    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r85r\" (UniqueName: \"kubernetes.io/projected/ef2ff06b-7835-4075-9346-7dc5ced29cf4-kube-api-access-2r85r\") pod \"payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv\" (UID: \"ef2ff06b-7835-4075-9346-7dc5ced29cf4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv"
Nov 02 01:49:57.961316 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:57.958366    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccbdec1a-9065-4b40-bf67-21d4f8c30e33-plugins-config-volume\") pod \"payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2\" (UID: \"ccbdec1a-9065-4b40-bf67-21d4f8c30e33\") " pod="openshift-ingress/payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2"
Nov 02 01:49:57.961316 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:57.958468    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t7db\" (UniqueName: \"kubernetes.io/projected/ccbdec1a-9065-4b40-bf67-21d4f8c30e33-kube-api-access-7t7db\") pod \"payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2\" (UID: \"ccbdec1a-9065-4b40-bf67-21d4f8c30e33\") " pod="openshift-ingress/payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2"
Nov 02 01:49:58.067354 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:58.065320    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccbdec1a-9065-4b40-bf67-21d4f8c30e33-plugins-config-volume\") pod \"payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2\" (UID: \"ccbdec1a-9065-4b40-bf67-21d4f8c30e33\") " pod="openshift-ingress/payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2"
Nov 02 01:49:58.067354 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:58.065426    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t7db\" (UniqueName: \"kubernetes.io/projected/ccbdec1a-9065-4b40-bf67-21d4f8c30e33-kube-api-access-7t7db\") pod \"payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2\" (UID: \"ccbdec1a-9065-4b40-bf67-21d4f8c30e33\") " pod="openshift-ingress/payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2"
Nov 02 01:49:58.084303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:58.067897    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccbdec1a-9065-4b40-bf67-21d4f8c30e33-plugins-config-volume\") pod \"payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2\" (UID: \"ccbdec1a-9065-4b40-bf67-21d4f8c30e33\") " pod="openshift-ingress/payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2"
Nov 02 01:49:58.096357 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:58.092602    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv"
Nov 02 01:49:58.188805 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:58.188769    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t7db\" (UniqueName: \"kubernetes.io/projected/ccbdec1a-9065-4b40-bf67-21d4f8c30e33-kube-api-access-7t7db\") pod \"payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2\" (UID: \"ccbdec1a-9065-4b40-bf67-21d4f8c30e33\") " pod="openshift-ingress/payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2"
Nov 02 01:49:58.407821 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:58.407786    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2"
Nov 02 01:49:58.542916 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:58.542876    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv"]
Nov 02 01:49:58.543423 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:49:58.543399    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef2ff06b_7835_4075_9346_7dc5ced29cf4.slice/crio-54776c43e49060a976bd6015f99a7778385393014e8a37a452d16f4e1fe628f8 WatchSource:0}: Error finding container 54776c43e49060a976bd6015f99a7778385393014e8a37a452d16f4e1fe628f8: Status 404 returned error can't find the container with id 54776c43e49060a976bd6015f99a7778385393014e8a37a452d16f4e1fe628f8
Nov 02 01:49:58.793165 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:49:58.793126    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbdec1a_9065_4b40_bf67_21d4f8c30e33.slice/crio-9fd88608f3d4560166ff80e6bb53eebc7f737837814d50fbb9402c016c3e1ac6 WatchSource:0}: Error finding container 9fd88608f3d4560166ff80e6bb53eebc7f737837814d50fbb9402c016c3e1ac6: Status 404 returned error can't find the container with id 9fd88608f3d4560166ff80e6bb53eebc7f737837814d50fbb9402c016c3e1ac6
Nov 02 01:49:58.794099 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:58.794071    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2"]
Nov 02 01:49:59.195843 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:59.195805    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv" event={"ID":"ef2ff06b-7835-4075-9346-7dc5ced29cf4","Type":"ContainerStarted","Data":"2b3fb118a3ab70f10756429b597e7c9627e94cdc1c0ce8b36c74b79163a7ee67"}
Nov 02 01:49:59.196064 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:59.195868    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv" event={"ID":"ef2ff06b-7835-4075-9346-7dc5ced29cf4","Type":"ContainerStarted","Data":"54776c43e49060a976bd6015f99a7778385393014e8a37a452d16f4e1fe628f8"}
Nov 02 01:49:59.196064 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:59.195894    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv"
Nov 02 01:49:59.197646 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:59.197620    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2" event={"ID":"ccbdec1a-9065-4b40-bf67-21d4f8c30e33","Type":"ContainerStarted","Data":"0fa86b5ba2b5c05d41d96be686b42796ccf9cf670c558b66457fe555b4fd104d"}
Nov 02 01:49:59.197802 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:59.197658    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2" event={"ID":"ccbdec1a-9065-4b40-bf67-21d4f8c30e33","Type":"ContainerStarted","Data":"9fd88608f3d4560166ff80e6bb53eebc7f737837814d50fbb9402c016c3e1ac6"}
Nov 02 01:49:59.197802 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:59.197721    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2"
Nov 02 01:49:59.312645 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:59.312579    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv" podStartSLOduration=2.312560749 podStartE2EDuration="2.312560749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:59.31248884 +0000 UTC m=+1464.216170956" watchObservedRunningTime="2026-11-02 01:49:59.312560749 +0000 UTC m=+1464.216242864"
Nov 02 01:49:59.440313 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:49:59.434265    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2" podStartSLOduration=2.4342433359999998 podStartE2EDuration="2.434243336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:59.433252712 +0000 UTC m=+1464.336934830" watchObservedRunningTime="2026-11-02 01:49:59.434243336 +0000 UTC m=+1464.337925447"
Nov 02 01:50:00.033211 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:00.033183    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-59d70a90-d54879985-jv9l7"
Nov 02 01:50:00.033762 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:00.033249    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd"
Nov 02 01:50:05.114608 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:05.114570    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr"
Nov 02 01:50:05.115089 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:05.114644    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp"
Nov 02 01:50:07.836320 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:07.834350    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt"]
Nov 02 01:50:07.873463 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:07.862567    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt"
Nov 02 01:50:07.873463 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:07.865969    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q"]
Nov 02 01:50:07.888305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:07.884424    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q"
Nov 02 01:50:07.948717 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:07.942572    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5fc56410\""
Nov 02 01:50:07.948717 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:07.942731    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5fc56410-dockercfg-zf6kb\""
Nov 02 01:50:07.948717 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:07.948507    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt"]
Nov 02 01:50:07.975327 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:07.971426    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q"]
Nov 02 01:50:08.036599 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:08.035459    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx4cb\" (UniqueName: \"kubernetes.io/projected/7392e99a-1aa8-48f9-ab30-4fea2eab88de-kube-api-access-cx4cb\") pod \"payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q\" (UID: \"7392e99a-1aa8-48f9-ab30-4fea2eab88de\") " pod="openshift-ingress/payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q"
Nov 02 01:50:08.037571 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:08.037545    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a0ddab3-c6be-44e4-800c-012437a54a2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt\" (UID: \"1a0ddab3-c6be-44e4-800c-012437a54a2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt"
Nov 02 01:50:08.037827 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:08.037803    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7392e99a-1aa8-48f9-ab30-4fea2eab88de-plugins-config-volume\") pod \"payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q\" (UID: \"7392e99a-1aa8-48f9-ab30-4fea2eab88de\") " pod="openshift-ingress/payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q"
Nov 02 01:50:08.038119 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:08.038101    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrrf8\" (UniqueName: \"kubernetes.io/projected/1a0ddab3-c6be-44e4-800c-012437a54a2e-kube-api-access-vrrf8\") pod \"payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt\" (UID: \"1a0ddab3-c6be-44e4-800c-012437a54a2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt"
Nov 02 01:50:08.165730 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:08.148339    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx4cb\" (UniqueName: \"kubernetes.io/projected/7392e99a-1aa8-48f9-ab30-4fea2eab88de-kube-api-access-cx4cb\") pod \"payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q\" (UID: \"7392e99a-1aa8-48f9-ab30-4fea2eab88de\") " pod="openshift-ingress/payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q"
Nov 02 01:50:08.165730 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:08.148504    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a0ddab3-c6be-44e4-800c-012437a54a2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt\" (UID: \"1a0ddab3-c6be-44e4-800c-012437a54a2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt"
Nov 02 01:50:08.165730 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:08.148572    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7392e99a-1aa8-48f9-ab30-4fea2eab88de-plugins-config-volume\") pod \"payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q\" (UID: \"7392e99a-1aa8-48f9-ab30-4fea2eab88de\") " pod="openshift-ingress/payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q"
Nov 02 01:50:08.165730 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:08.148740    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrrf8\" (UniqueName: \"kubernetes.io/projected/1a0ddab3-c6be-44e4-800c-012437a54a2e-kube-api-access-vrrf8\") pod \"payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt\" (UID: \"1a0ddab3-c6be-44e4-800c-012437a54a2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt"
Nov 02 01:50:08.190986 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:08.190951    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a0ddab3-c6be-44e4-800c-012437a54a2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt\" (UID: \"1a0ddab3-c6be-44e4-800c-012437a54a2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt"
Nov 02 01:50:08.194045 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:08.193989    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7392e99a-1aa8-48f9-ab30-4fea2eab88de-plugins-config-volume\") pod \"payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q\" (UID: \"7392e99a-1aa8-48f9-ab30-4fea2eab88de\") " pod="openshift-ingress/payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q"
Nov 02 01:50:08.338604 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:08.282039    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrrf8\" (UniqueName: \"kubernetes.io/projected/1a0ddab3-c6be-44e4-800c-012437a54a2e-kube-api-access-vrrf8\") pod \"payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt\" (UID: \"1a0ddab3-c6be-44e4-800c-012437a54a2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt"
Nov 02 01:50:08.338604 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:08.297585    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt"
Nov 02 01:50:08.367961 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:08.359617    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx4cb\" (UniqueName: \"kubernetes.io/projected/7392e99a-1aa8-48f9-ab30-4fea2eab88de-kube-api-access-cx4cb\") pod \"payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q\" (UID: \"7392e99a-1aa8-48f9-ab30-4fea2eab88de\") " pod="openshift-ingress/payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q"
Nov 02 01:50:08.657125 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:08.656090    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q"
Nov 02 01:50:08.690661 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:08.690609    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:50:09.372544 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:09.372506    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt"]
Nov 02 01:50:09.445219 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:09.444805    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q"]
Nov 02 01:50:09.446762 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:50:09.445303    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7392e99a_1aa8_48f9_ab30_4fea2eab88de.slice/crio-38edba4917ee3ed4decc038df2172e853ac02a7e9087213a10a953dbf8a0897c WatchSource:0}: Error finding container 38edba4917ee3ed4decc038df2172e853ac02a7e9087213a10a953dbf8a0897c: Status 404 returned error can't find the container with id 38edba4917ee3ed4decc038df2172e853ac02a7e9087213a10a953dbf8a0897c
Nov 02 01:50:09.700201 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:09.700086    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q" event={"ID":"7392e99a-1aa8-48f9-ab30-4fea2eab88de","Type":"ContainerStarted","Data":"38edba4917ee3ed4decc038df2172e853ac02a7e9087213a10a953dbf8a0897c"}
Nov 02 01:50:09.703628 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:09.703580    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt" event={"ID":"1a0ddab3-c6be-44e4-800c-012437a54a2e","Type":"ContainerStarted","Data":"ab7c062a6869c0c1efc22b5373be12c6613d4820cb1b9c0e5dfae1118f0c698c"}
Nov 02 01:50:09.703903 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:09.703865    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt" event={"ID":"1a0ddab3-c6be-44e4-800c-012437a54a2e","Type":"ContainerStarted","Data":"592066d617951d2c8c6679033715c7c69c64b8a719b0b0f982b601ba5052e2cd"}
Nov 02 01:50:09.727197 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:09.727160    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt"
Nov 02 01:50:09.894133 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:09.889213    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt" podStartSLOduration=2.889191146 podStartE2EDuration="2.889191146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:50:09.847525315 +0000 UTC m=+1474.751207429" watchObservedRunningTime="2026-11-02 01:50:09.889191146 +0000 UTC m=+1474.792873261"
Nov 02 01:50:10.246324 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:10.242500    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2"
Nov 02 01:50:10.247550 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:10.247519    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv"
Nov 02 01:50:10.884962 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:10.884854    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q" event={"ID":"7392e99a-1aa8-48f9-ab30-4fea2eab88de","Type":"ContainerStarted","Data":"ddbce4175bb3059478807e91aac4e5dff685c9ef7decdfa9f47a86c543348254"}
Nov 02 01:50:10.884962 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:10.884911    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q"
Nov 02 01:50:11.036934 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:11.036869    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q" podStartSLOduration=4.036850288 podStartE2EDuration="4.036850288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:50:11.019961108 +0000 UTC m=+1475.923643223" watchObservedRunningTime="2026-11-02 01:50:11.036850288 +0000 UTC m=+1475.940532403"
Nov 02 01:50:12.343462 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:12.343414    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:50:12.927307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:12.926333    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 02 01:50:12.927307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:12.926633    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Nov 02 01:50:12.930589 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:12.930538    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": context deadline exceeded"
Nov 02 01:50:12.934992 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:12.934952    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:50:12.983939 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:12.983833    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:50:13.146474 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:13.146435    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:50:13.201584 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:13.201472    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:50:15.296293 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.296243    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"]
Nov 02 01:50:15.302667 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.302638    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.412679 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.412643    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"]
Nov 02 01:50:15.430092 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.430062    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.430272 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.430177    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.430435 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.430339    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.430435 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.430409    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.430552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.430528    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a02ec44c-7c77-41b5-967e-b83c696ee76f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.430691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.430672    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.430856 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.430838    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.430935 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.430878    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92qg5\" (UniqueName: \"kubernetes.io/projected/a02ec44c-7c77-41b5-967e-b83c696ee76f-kube-api-access-92qg5\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.430998 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.430935    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.532112 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.532079    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.532356 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.532129    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a02ec44c-7c77-41b5-967e-b83c696ee76f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.532356 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.532207    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.532489 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.532370    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.532489 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.532404    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92qg5\" (UniqueName: \"kubernetes.io/projected/a02ec44c-7c77-41b5-967e-b83c696ee76f-kube-api-access-92qg5\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.532489 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.532432    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.532667 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.532509    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.532667 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.532593    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.532765 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.532671    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.532765 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.532708    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.532965 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.532909    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.532965 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.532954    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.533209 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.533046    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.533209 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.533074    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a02ec44c-7c77-41b5-967e-b83c696ee76f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.534714 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.534694    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.535449 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.535433    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.593530 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.593493    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.593722 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.593707    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92qg5\" (UniqueName: \"kubernetes.io/projected/a02ec44c-7c77-41b5-967e-b83c696ee76f-kube-api-access-92qg5\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-nqz7s\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:15.619557 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:15.619525    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:16.094316 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:16.091308    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"]
Nov 02 01:50:16.094316 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:50:16.091823    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda02ec44c_7c77_41b5_967e_b83c696ee76f.slice/crio-a4aabef8c3aae08170f3ca6651629db144011efca8dd69ddf437c10218f0632e WatchSource:0}: Error finding container a4aabef8c3aae08170f3ca6651629db144011efca8dd69ddf437c10218f0632e: Status 404 returned error can't find the container with id a4aabef8c3aae08170f3ca6651629db144011efca8dd69ddf437c10218f0632e
Nov 02 01:50:16.109914 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:16.095933    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:50:16.109914 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:16.096019    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:50:16.109914 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:16.096059    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:50:17.058860 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:17.058815    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s" event={"ID":"a02ec44c-7c77-41b5-967e-b83c696ee76f","Type":"ContainerStarted","Data":"930f1aeb207c71c12a4e2dc9df3ef4778ecf24aa9012bdd793df0002e6f3e848"}
Nov 02 01:50:17.059251 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:17.058920    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s" event={"ID":"a02ec44c-7c77-41b5-967e-b83c696ee76f","Type":"ContainerStarted","Data":"a4aabef8c3aae08170f3ca6651629db144011efca8dd69ddf437c10218f0632e"}
Nov 02 01:50:17.234299 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:17.234229    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s" podStartSLOduration=2.234212833 podStartE2EDuration="2.234212833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:50:17.230339874 +0000 UTC m=+1482.134021989" watchObservedRunningTime="2026-11-02 01:50:17.234212833 +0000 UTC m=+1482.137894947"
Nov 02 01:50:17.629776 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:17.629570    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:17.633110 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:17.632526    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 02 01:50:17.633110 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:17.632604    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s" podUID="a02ec44c-7c77-41b5-967e-b83c696ee76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 02 01:50:18.620883 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:18.620841    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 02 01:50:18.621402 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:18.620917    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s" podUID="a02ec44c-7c77-41b5-967e-b83c696ee76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 02 01:50:19.620719 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:19.620679    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 02 01:50:19.620913 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:19.620747    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s" podUID="a02ec44c-7c77-41b5-967e-b83c696ee76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 02 01:50:20.620554 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:20.620519    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 02 01:50:20.620735 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:20.620591    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s" podUID="a02ec44c-7c77-41b5-967e-b83c696ee76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 02 01:50:21.620700 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:21.620661    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 02 01:50:21.621264 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:21.620728    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s" podUID="a02ec44c-7c77-41b5-967e-b83c696ee76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 02 01:50:21.949251 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:21.949172    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt"
Nov 02 01:50:22.003644 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:22.003257    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q"
Nov 02 01:50:22.620629 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:22.620591    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 02 01:50:22.620835 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:22.620665    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s" podUID="a02ec44c-7c77-41b5-967e-b83c696ee76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 02 01:50:23.145103 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:23.145059    5639 generic.go:356] "Generic (PLEG): container finished" podID="8b71baa1-9d31-48de-a3ec-379f6f5eae37" containerID="5107a59619749632b90e634b66f0b7ba6535efb394ae63ff6752b6f76fc7bbe0" exitCode=0
Nov 02 01:50:23.145340 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:23.145105    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5" event={"ID":"8b71baa1-9d31-48de-a3ec-379f6f5eae37","Type":"ContainerDied","Data":"5107a59619749632b90e634b66f0b7ba6535efb394ae63ff6752b6f76fc7bbe0"}
Nov 02 01:50:23.145659 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:23.145631    5639 scope.go:122] "RemoveContainer" containerID="5107a59619749632b90e634b66f0b7ba6535efb394ae63ff6752b6f76fc7bbe0"
Nov 02 01:50:23.147372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:23.147344    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:50:23.147458 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:23.147431    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:50:23.147517 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:23.147465    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:50:23.302567 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:23.302525    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:50:23.620180 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:23.620142    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 02 01:50:23.620422 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:23.620205    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s" podUID="a02ec44c-7c77-41b5-967e-b83c696ee76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 02 01:50:24.158077 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:24.158037    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5" event={"ID":"8b71baa1-9d31-48de-a3ec-379f6f5eae37","Type":"ContainerStarted","Data":"fc7d3e2d55b46e5000bb6ce2b590e7dd61e5f93b00b9458c0d8fca23b64fad30"}
Nov 02 01:50:24.620247 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:24.620209    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 02 01:50:24.620467 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:24.620309    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s" podUID="a02ec44c-7c77-41b5-967e-b83c696ee76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 02 01:50:24.891818 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:24.891313    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:50:24.891818 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:24.891436    5639 patch_prober.go:29] interesting pod/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 01:50:24.891818 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:24.891478    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5" podUID="8b71baa1-9d31-48de-a3ec-379f6f5eae37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 01:50:25.624252 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:25.624220    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:25.624867 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:25.624849    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:25.625520 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:25.625501    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 01:50:25.889470 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:25.889380    5639 patch_prober.go:29] interesting pod/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 01:50:25.889743 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:25.889718    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5" podUID="8b71baa1-9d31-48de-a3ec-379f6f5eae37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 01:50:26.889870 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:26.889830    5639 patch_prober.go:29] interesting pod/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 01:50:26.890508 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:26.890475    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5" podUID="8b71baa1-9d31-48de-a3ec-379f6f5eae37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 01:50:27.526456 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:50:27.526419    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:50:27.889808 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:27.889774    5639 patch_prober.go:29] interesting pod/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 01:50:27.889999 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:27.889831    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5" podUID="8b71baa1-9d31-48de-a3ec-379f6f5eae37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 01:50:28.890762 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:28.890729    5639 patch_prober.go:29] interesting pod/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 01:50:28.891252 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:28.890812    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5" podUID="8b71baa1-9d31-48de-a3ec-379f6f5eae37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 01:50:29.894468 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:29.894438    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:50:29.894960 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:29.894939    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:50:29.895789 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:29.895773    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:50:35.943974 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:35.943945    5639 scope.go:122] "RemoveContainer" containerID="bd7834aab877d34ff0a9911a7523a202f152fca18427f8e4fad545318cc964dd"
Nov 02 01:50:50.973381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:50.973347    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59d70a90-5thbz"]
Nov 02 01:50:51.076304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:51.060831    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59d70a90-5thbz"]
Nov 02 01:50:51.076304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:51.060953    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59d70a90-5thbz"
Nov 02 01:50:51.116322 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:51.115522    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:50:51.116322 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:51.115620    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:50:51.116322 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:51.115651    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:50:51.116322 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:51.115718    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:50:51.150507 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:51.150459    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:50:51.198999 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:51.198959    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jms4k\" (UniqueName: \"kubernetes.io/projected/6c0947ba-7328-4226-8fe2-8a781d8c9748-kube-api-access-jms4k\") pod \"maas-api-revoke-keys-e2e-ait-59d70a90-5thbz\" (UID: \"6c0947ba-7328-4226-8fe2-8a781d8c9748\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59d70a90-5thbz"
Nov 02 01:50:51.199202 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:51.199092    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c0947ba-7328-4226-8fe2-8a781d8c9748-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-59d70a90-5thbz\" (UID: \"6c0947ba-7328-4226-8fe2-8a781d8c9748\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59d70a90-5thbz"
Nov 02 01:50:51.199202 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:51.199152    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c0947ba-7328-4226-8fe2-8a781d8c9748-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-59d70a90-5thbz\" (UID: \"6c0947ba-7328-4226-8fe2-8a781d8c9748\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59d70a90-5thbz"
Nov 02 01:50:51.312359 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:51.310088    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jms4k\" (UniqueName: \"kubernetes.io/projected/6c0947ba-7328-4226-8fe2-8a781d8c9748-kube-api-access-jms4k\") pod \"maas-api-revoke-keys-e2e-ait-59d70a90-5thbz\" (UID: \"6c0947ba-7328-4226-8fe2-8a781d8c9748\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59d70a90-5thbz"
Nov 02 01:50:51.312359 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:51.310219    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c0947ba-7328-4226-8fe2-8a781d8c9748-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-59d70a90-5thbz\" (UID: \"6c0947ba-7328-4226-8fe2-8a781d8c9748\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59d70a90-5thbz"
Nov 02 01:50:51.312359 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:51.310343    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c0947ba-7328-4226-8fe2-8a781d8c9748-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-59d70a90-5thbz\" (UID: \"6c0947ba-7328-4226-8fe2-8a781d8c9748\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59d70a90-5thbz"
Nov 02 01:50:51.312359 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:51.311195    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c0947ba-7328-4226-8fe2-8a781d8c9748-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-59d70a90-5thbz\" (UID: \"6c0947ba-7328-4226-8fe2-8a781d8c9748\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59d70a90-5thbz"
Nov 02 01:50:51.341312 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:51.340220    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c0947ba-7328-4226-8fe2-8a781d8c9748-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-59d70a90-5thbz\" (UID: \"6c0947ba-7328-4226-8fe2-8a781d8c9748\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59d70a90-5thbz"
Nov 02 01:50:51.388360 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:51.382643    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jms4k\" (UniqueName: \"kubernetes.io/projected/6c0947ba-7328-4226-8fe2-8a781d8c9748-kube-api-access-jms4k\") pod \"maas-api-revoke-keys-e2e-ait-59d70a90-5thbz\" (UID: \"6c0947ba-7328-4226-8fe2-8a781d8c9748\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59d70a90-5thbz"
Nov 02 01:50:51.400852 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:51.400564    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59d70a90-5thbz"
Nov 02 01:50:52.338361 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:50:52.338251    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c0947ba_7328_4226_8fe2_8a781d8c9748.slice/crio-575db8c2f36a320d2a0779fef2461758d27aad16e4e9719053ca8537d103222a WatchSource:0}: Error finding container 575db8c2f36a320d2a0779fef2461758d27aad16e4e9719053ca8537d103222a: Status 404 returned error can't find the container with id 575db8c2f36a320d2a0779fef2461758d27aad16e4e9719053ca8537d103222a
Nov 02 01:50:52.338915 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:52.338714    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59d70a90-5thbz"]
Nov 02 01:50:52.467433 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:52.467391    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59d70a90-5thbz" event={"ID":"6c0947ba-7328-4226-8fe2-8a781d8c9748","Type":"ContainerStarted","Data":"575db8c2f36a320d2a0779fef2461758d27aad16e4e9719053ca8537d103222a"}
Nov 02 01:50:53.517319 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:53.504241    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59d70a90-5thbz" event={"ID":"6c0947ba-7328-4226-8fe2-8a781d8c9748","Type":"ContainerStarted","Data":"7e3be3ffe8c7b79bf2c1ae1798d1e45ae6c9685172b3724bebb0c03fe4afbfd6"}
Nov 02 01:50:53.710312 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:53.709834    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59d70a90-5thbz" podStartSLOduration=3.70981301 podStartE2EDuration="3.70981301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:50:53.645254112 +0000 UTC m=+1518.548936230" watchObservedRunningTime="2026-11-02 01:50:53.70981301 +0000 UTC m=+1518.613495127"
Nov 02 01:50:54.514472 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:54.514437    5639 generic.go:356] "Generic (PLEG): container finished" podID="6c0947ba-7328-4226-8fe2-8a781d8c9748" containerID="7e3be3ffe8c7b79bf2c1ae1798d1e45ae6c9685172b3724bebb0c03fe4afbfd6" exitCode=0
Nov 02 01:50:54.514684 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:54.514525    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59d70a90-5thbz" event={"ID":"6c0947ba-7328-4226-8fe2-8a781d8c9748","Type":"ContainerDied","Data":"7e3be3ffe8c7b79bf2c1ae1798d1e45ae6c9685172b3724bebb0c03fe4afbfd6"}
Nov 02 01:50:55.685882 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:55.685855    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59d70a90-5thbz"
Nov 02 01:50:55.804399 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:55.803135    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c0947ba-7328-4226-8fe2-8a781d8c9748-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c0947ba-7328-4226-8fe2-8a781d8c9748-maas-api-xks-ca\") pod \"6c0947ba-7328-4226-8fe2-8a781d8c9748\" (UID: \"6c0947ba-7328-4226-8fe2-8a781d8c9748\") "
Nov 02 01:50:55.804399 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:55.803442    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c0947ba-7328-4226-8fe2-8a781d8c9748-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c0947ba-7328-4226-8fe2-8a781d8c9748-maas-api-openshift-ca\") pod \"6c0947ba-7328-4226-8fe2-8a781d8c9748\" (UID: \"6c0947ba-7328-4226-8fe2-8a781d8c9748\") "
Nov 02 01:50:55.804399 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:55.803581    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c0947ba-7328-4226-8fe2-8a781d8c9748-kube-api-access-jms4k\" (UniqueName: \"kubernetes.io/projected/6c0947ba-7328-4226-8fe2-8a781d8c9748-kube-api-access-jms4k\") pod \"6c0947ba-7328-4226-8fe2-8a781d8c9748\" (UID: \"6c0947ba-7328-4226-8fe2-8a781d8c9748\") "
Nov 02 01:50:55.804399 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:55.804198    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c0947ba-7328-4226-8fe2-8a781d8c9748-maas-api-openshift-ca" pod "6c0947ba-7328-4226-8fe2-8a781d8c9748" (UID: "6c0947ba-7328-4226-8fe2-8a781d8c9748"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:50:55.804399 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:55.804315    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c0947ba-7328-4226-8fe2-8a781d8c9748-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:50:55.817505 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:55.817466    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c0947ba-7328-4226-8fe2-8a781d8c9748-maas-api-xks-ca" pod "6c0947ba-7328-4226-8fe2-8a781d8c9748" (UID: "6c0947ba-7328-4226-8fe2-8a781d8c9748"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:50:55.819047 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:55.818993    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c0947ba-7328-4226-8fe2-8a781d8c9748-kube-api-access-jms4k" pod "6c0947ba-7328-4226-8fe2-8a781d8c9748" (UID: "6c0947ba-7328-4226-8fe2-8a781d8c9748"). InnerVolumeSpecName "kube-api-access-jms4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:50:55.906254 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:55.906222    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jms4k\" (UniqueName: \"kubernetes.io/projected/6c0947ba-7328-4226-8fe2-8a781d8c9748-kube-api-access-jms4k\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:50:55.906254 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:55.906253    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c0947ba-7328-4226-8fe2-8a781d8c9748-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:50:56.531945 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:56.531914    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59d70a90-5thbz"
Nov 02 01:50:56.532163 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:56.531947    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59d70a90-5thbz" event={"ID":"6c0947ba-7328-4226-8fe2-8a781d8c9748","Type":"ContainerDied","Data":"575db8c2f36a320d2a0779fef2461758d27aad16e4e9719053ca8537d103222a"}
Nov 02 01:50:56.532163 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:56.532020    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="575db8c2f36a320d2a0779fef2461758d27aad16e4e9719053ca8537d103222a"
Nov 02 01:50:57.423027 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:50:57.422990    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:50:58.245625 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.245597    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd723454-rgtxn"]
Nov 02 01:50:58.246432 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.246415    5639 cpu_manager.go:414] "containerMap: removing container" podUID="6c0947ba-7328-4226-8fe2-8a781d8c9748" containerName="revoke-keys"
Nov 02 01:50:58.246505 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.246433    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c0947ba-7328-4226-8fe2-8a781d8c9748" containerName="revoke-keys"
Nov 02 01:50:58.246590 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.246581    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c0947ba-7328-4226-8fe2-8a781d8c9748" containerName="revoke-keys"
Nov 02 01:50:58.249397 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.249381    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd723454-rgtxn"
Nov 02 01:50:58.277786 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.277759    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddc019a8-a65c-4f8a-9d98-a638232a10ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-fd723454-rgtxn\" (UID: \"ddc019a8-a65c-4f8a-9d98-a638232a10ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd723454-rgtxn"
Nov 02 01:50:58.277911 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.277835    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfs9c\" (UniqueName: \"kubernetes.io/projected/ddc019a8-a65c-4f8a-9d98-a638232a10ec-kube-api-access-xfs9c\") pod \"maas-api-revoke-keys-e2e-derive-fd723454-rgtxn\" (UID: \"ddc019a8-a65c-4f8a-9d98-a638232a10ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd723454-rgtxn"
Nov 02 01:50:58.277961 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.277947    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddc019a8-a65c-4f8a-9d98-a638232a10ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-fd723454-rgtxn\" (UID: \"ddc019a8-a65c-4f8a-9d98-a638232a10ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd723454-rgtxn"
Nov 02 01:50:58.302648 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.302622    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:50:58.308595 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.308566    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd723454-rgtxn"]
Nov 02 01:50:58.350120 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.350093    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:50:58.350330 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.350146    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:50:58.350330 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.350171    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:50:58.379128 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.379092    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddc019a8-a65c-4f8a-9d98-a638232a10ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-fd723454-rgtxn\" (UID: \"ddc019a8-a65c-4f8a-9d98-a638232a10ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd723454-rgtxn"
Nov 02 01:50:58.379299 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.379177    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfs9c\" (UniqueName: \"kubernetes.io/projected/ddc019a8-a65c-4f8a-9d98-a638232a10ec-kube-api-access-xfs9c\") pod \"maas-api-revoke-keys-e2e-derive-fd723454-rgtxn\" (UID: \"ddc019a8-a65c-4f8a-9d98-a638232a10ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd723454-rgtxn"
Nov 02 01:50:58.379342 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.379305    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddc019a8-a65c-4f8a-9d98-a638232a10ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-fd723454-rgtxn\" (UID: \"ddc019a8-a65c-4f8a-9d98-a638232a10ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd723454-rgtxn"
Nov 02 01:50:58.379896 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.379874    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddc019a8-a65c-4f8a-9d98-a638232a10ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-fd723454-rgtxn\" (UID: \"ddc019a8-a65c-4f8a-9d98-a638232a10ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd723454-rgtxn"
Nov 02 01:50:58.381637 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.381622    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddc019a8-a65c-4f8a-9d98-a638232a10ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-fd723454-rgtxn\" (UID: \"ddc019a8-a65c-4f8a-9d98-a638232a10ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd723454-rgtxn"
Nov 02 01:50:58.439781 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.439742    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfs9c\" (UniqueName: \"kubernetes.io/projected/ddc019a8-a65c-4f8a-9d98-a638232a10ec-kube-api-access-xfs9c\") pod \"maas-api-revoke-keys-e2e-derive-fd723454-rgtxn\" (UID: \"ddc019a8-a65c-4f8a-9d98-a638232a10ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd723454-rgtxn"
Nov 02 01:50:58.564395 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.564317    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd723454-rgtxn"
Nov 02 01:50:58.658676 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.658648    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2821b375-j8mr8"]
Nov 02 01:50:58.662558 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.662537    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2821b375-j8mr8"
Nov 02 01:50:58.684956 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.683719    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56b5b049-867f-4f13-90e5-d2b280044f2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2821b375-j8mr8\" (UID: \"56b5b049-867f-4f13-90e5-d2b280044f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2821b375-j8mr8"
Nov 02 01:50:58.684956 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.684336    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns9qr\" (UniqueName: \"kubernetes.io/projected/56b5b049-867f-4f13-90e5-d2b280044f2d-kube-api-access-ns9qr\") pod \"maas-api-revoke-keys-e2e-ait-2821b375-j8mr8\" (UID: \"56b5b049-867f-4f13-90e5-d2b280044f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2821b375-j8mr8"
Nov 02 01:50:58.684956 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.684388    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56b5b049-867f-4f13-90e5-d2b280044f2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2821b375-j8mr8\" (UID: \"56b5b049-867f-4f13-90e5-d2b280044f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2821b375-j8mr8"
Nov 02 01:50:58.723428 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.723397    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2821b375-j8mr8"]
Nov 02 01:50:58.785937 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.785899    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56b5b049-867f-4f13-90e5-d2b280044f2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2821b375-j8mr8\" (UID: \"56b5b049-867f-4f13-90e5-d2b280044f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2821b375-j8mr8"
Nov 02 01:50:58.786128 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.786116    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns9qr\" (UniqueName: \"kubernetes.io/projected/56b5b049-867f-4f13-90e5-d2b280044f2d-kube-api-access-ns9qr\") pod \"maas-api-revoke-keys-e2e-ait-2821b375-j8mr8\" (UID: \"56b5b049-867f-4f13-90e5-d2b280044f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2821b375-j8mr8"
Nov 02 01:50:58.786173 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.786139    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56b5b049-867f-4f13-90e5-d2b280044f2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2821b375-j8mr8\" (UID: \"56b5b049-867f-4f13-90e5-d2b280044f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2821b375-j8mr8"
Nov 02 01:50:58.786577 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.786554    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56b5b049-867f-4f13-90e5-d2b280044f2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2821b375-j8mr8\" (UID: \"56b5b049-867f-4f13-90e5-d2b280044f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2821b375-j8mr8"
Nov 02 01:50:58.788892 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.788866    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56b5b049-867f-4f13-90e5-d2b280044f2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2821b375-j8mr8\" (UID: \"56b5b049-867f-4f13-90e5-d2b280044f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2821b375-j8mr8"
Nov 02 01:50:58.847182 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.847150    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns9qr\" (UniqueName: \"kubernetes.io/projected/56b5b049-867f-4f13-90e5-d2b280044f2d-kube-api-access-ns9qr\") pod \"maas-api-revoke-keys-e2e-ait-2821b375-j8mr8\" (UID: \"56b5b049-867f-4f13-90e5-d2b280044f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2821b375-j8mr8"
Nov 02 01:50:58.945562 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:50:58.945531    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddc019a8_a65c_4f8a_9d98_a638232a10ec.slice/crio-3d45966600b744451e10ad6538334a81417a197d80d308833e77b71f8ede0c86 WatchSource:0}: Error finding container 3d45966600b744451e10ad6538334a81417a197d80d308833e77b71f8ede0c86: Status 404 returned error can't find the container with id 3d45966600b744451e10ad6538334a81417a197d80d308833e77b71f8ede0c86
Nov 02 01:50:58.946878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.946860    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd723454-rgtxn"]
Nov 02 01:50:58.980656 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:58.980634    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2821b375-j8mr8"
Nov 02 01:50:59.362118 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:50:59.362087    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b5b049_867f_4f13_90e5_d2b280044f2d.slice/crio-0aa69326df70f21f5dfc69ffca89b205736fd21506935bfa7bf04e8a478849ed WatchSource:0}: Error finding container 0aa69326df70f21f5dfc69ffca89b205736fd21506935bfa7bf04e8a478849ed: Status 404 returned error can't find the container with id 0aa69326df70f21f5dfc69ffca89b205736fd21506935bfa7bf04e8a478849ed
Nov 02 01:50:59.365716 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:59.365688    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2821b375-j8mr8"]
Nov 02 01:50:59.559598 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:59.559501    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd723454-rgtxn" event={"ID":"ddc019a8-a65c-4f8a-9d98-a638232a10ec","Type":"ContainerStarted","Data":"97b7b40d3a72ccf1b6a8c6d2019a6ed0e4a7d34d103e00ee06691209b3b2637b"}
Nov 02 01:50:59.559598 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:59.559563    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd723454-rgtxn" event={"ID":"ddc019a8-a65c-4f8a-9d98-a638232a10ec","Type":"ContainerStarted","Data":"3d45966600b744451e10ad6538334a81417a197d80d308833e77b71f8ede0c86"}
Nov 02 01:50:59.561655 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:59.561623    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2821b375-j8mr8" event={"ID":"56b5b049-867f-4f13-90e5-d2b280044f2d","Type":"ContainerStarted","Data":"0aa69326df70f21f5dfc69ffca89b205736fd21506935bfa7bf04e8a478849ed"}
Nov 02 01:50:59.679614 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:50:59.679562    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd723454-rgtxn" podStartSLOduration=1.679544785 podStartE2EDuration="1.679544785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:50:59.675957986 +0000 UTC m=+1524.579640099" watchObservedRunningTime="2026-11-02 01:50:59.679544785 +0000 UTC m=+1524.583226898"
Nov 02 01:51:00.239132 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:00.239055    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31955345-qdbvw"]
Nov 02 01:51:00.242567 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:00.242551    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31955345-qdbvw"
Nov 02 01:51:00.300961 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:00.300930    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31955345-qdbvw"]
Nov 02 01:51:00.310836 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:00.310802    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hchfm\" (UniqueName: \"kubernetes.io/projected/3f24c71c-bafc-46fa-b9f4-6ee006c40953-kube-api-access-hchfm\") pod \"maas-api-revoke-sub-31955345-qdbvw\" (UID: \"3f24c71c-bafc-46fa-b9f4-6ee006c40953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31955345-qdbvw"
Nov 02 01:51:00.311017 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:00.310927    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f24c71c-bafc-46fa-b9f4-6ee006c40953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31955345-qdbvw\" (UID: \"3f24c71c-bafc-46fa-b9f4-6ee006c40953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31955345-qdbvw"
Nov 02 01:51:00.311177 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:00.311145    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f24c71c-bafc-46fa-b9f4-6ee006c40953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31955345-qdbvw\" (UID: \"3f24c71c-bafc-46fa-b9f4-6ee006c40953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31955345-qdbvw"
Nov 02 01:51:00.413076 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:00.413045    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hchfm\" (UniqueName: \"kubernetes.io/projected/3f24c71c-bafc-46fa-b9f4-6ee006c40953-kube-api-access-hchfm\") pod \"maas-api-revoke-sub-31955345-qdbvw\" (UID: \"3f24c71c-bafc-46fa-b9f4-6ee006c40953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31955345-qdbvw"
Nov 02 01:51:00.413250 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:00.413139    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f24c71c-bafc-46fa-b9f4-6ee006c40953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31955345-qdbvw\" (UID: \"3f24c71c-bafc-46fa-b9f4-6ee006c40953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31955345-qdbvw"
Nov 02 01:51:00.413387 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:00.413307    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f24c71c-bafc-46fa-b9f4-6ee006c40953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31955345-qdbvw\" (UID: \"3f24c71c-bafc-46fa-b9f4-6ee006c40953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31955345-qdbvw"
Nov 02 01:51:00.413844 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:00.413822    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f24c71c-bafc-46fa-b9f4-6ee006c40953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31955345-qdbvw\" (UID: \"3f24c71c-bafc-46fa-b9f4-6ee006c40953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31955345-qdbvw"
Nov 02 01:51:00.415746 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:00.415723    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f24c71c-bafc-46fa-b9f4-6ee006c40953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31955345-qdbvw\" (UID: \"3f24c71c-bafc-46fa-b9f4-6ee006c40953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31955345-qdbvw"
Nov 02 01:51:00.473712 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:00.473681    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hchfm\" (UniqueName: \"kubernetes.io/projected/3f24c71c-bafc-46fa-b9f4-6ee006c40953-kube-api-access-hchfm\") pod \"maas-api-revoke-sub-31955345-qdbvw\" (UID: \"3f24c71c-bafc-46fa-b9f4-6ee006c40953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31955345-qdbvw"
Nov 02 01:51:00.555426 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:00.555345    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31955345-qdbvw"
Nov 02 01:51:00.580791 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:00.580752    5639 generic.go:356] "Generic (PLEG): container finished" podID="56b5b049-867f-4f13-90e5-d2b280044f2d" containerID="b9f62d5475436886ec53834c5e6ec9f9255400d3ea6c17c71fb071298efe928d" exitCode=0
Nov 02 01:51:00.581194 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:00.580801    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2821b375-j8mr8" event={"ID":"56b5b049-867f-4f13-90e5-d2b280044f2d","Type":"ContainerDied","Data":"b9f62d5475436886ec53834c5e6ec9f9255400d3ea6c17c71fb071298efe928d"}
Nov 02 01:51:00.582823 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:00.582800    5639 generic.go:356] "Generic (PLEG): container finished" podID="ddc019a8-a65c-4f8a-9d98-a638232a10ec" containerID="97b7b40d3a72ccf1b6a8c6d2019a6ed0e4a7d34d103e00ee06691209b3b2637b" exitCode=0
Nov 02 01:51:00.582939 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:00.582853    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd723454-rgtxn" event={"ID":"ddc019a8-a65c-4f8a-9d98-a638232a10ec","Type":"ContainerDied","Data":"97b7b40d3a72ccf1b6a8c6d2019a6ed0e4a7d34d103e00ee06691209b3b2637b"}
Nov 02 01:51:00.950878 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:51:00.950850    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f24c71c_bafc_46fa_b9f4_6ee006c40953.slice/crio-2f293d276534971c726c4744cb948c38ca4b61261ea5bae49f75dd18e6d02484 WatchSource:0}: Error finding container 2f293d276534971c726c4744cb948c38ca4b61261ea5bae49f75dd18e6d02484: Status 404 returned error can't find the container with id 2f293d276534971c726c4744cb948c38ca4b61261ea5bae49f75dd18e6d02484
Nov 02 01:51:00.953727 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:00.953703    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31955345-qdbvw"]
Nov 02 01:51:01.593704 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:01.593672    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31955345-qdbvw" event={"ID":"3f24c71c-bafc-46fa-b9f4-6ee006c40953","Type":"ContainerStarted","Data":"16d9321f8e6b42fcd33c92c0be4a017e049f75d3dd760ac703b1f2326a1cbb0e"}
Nov 02 01:51:01.594124 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:01.593720    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31955345-qdbvw" event={"ID":"3f24c71c-bafc-46fa-b9f4-6ee006c40953","Type":"ContainerStarted","Data":"2f293d276534971c726c4744cb948c38ca4b61261ea5bae49f75dd18e6d02484"}
Nov 02 01:51:01.719256 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:01.719194    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31955345-qdbvw" podStartSLOduration=1.719171797 podStartE2EDuration="1.719171797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:51:01.708706826 +0000 UTC m=+1526.612388942" watchObservedRunningTime="2026-11-02 01:51:01.719171797 +0000 UTC m=+1526.622853913"
Nov 02 01:51:01.799141 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:01.799112    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2821b375-j8mr8"
Nov 02 01:51:01.803763 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:01.803741    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd723454-rgtxn"
Nov 02 01:51:01.839718 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:01.839692    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56b5b049-867f-4f13-90e5-d2b280044f2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56b5b049-867f-4f13-90e5-d2b280044f2d-maas-api-xks-ca\") pod \"56b5b049-867f-4f13-90e5-d2b280044f2d\" (UID: \"56b5b049-867f-4f13-90e5-d2b280044f2d\") "
Nov 02 01:51:01.839855 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:01.839727    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56b5b049-867f-4f13-90e5-d2b280044f2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56b5b049-867f-4f13-90e5-d2b280044f2d-maas-api-openshift-ca\") pod \"56b5b049-867f-4f13-90e5-d2b280044f2d\" (UID: \"56b5b049-867f-4f13-90e5-d2b280044f2d\") "
Nov 02 01:51:01.839855 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:01.839807    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56b5b049-867f-4f13-90e5-d2b280044f2d-kube-api-access-ns9qr\" (UniqueName: \"kubernetes.io/projected/56b5b049-867f-4f13-90e5-d2b280044f2d-kube-api-access-ns9qr\") pod \"56b5b049-867f-4f13-90e5-d2b280044f2d\" (UID: \"56b5b049-867f-4f13-90e5-d2b280044f2d\") "
Nov 02 01:51:01.840056 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:01.840038    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddc019a8-a65c-4f8a-9d98-a638232a10ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddc019a8-a65c-4f8a-9d98-a638232a10ec-maas-api-xks-ca\") pod \"ddc019a8-a65c-4f8a-9d98-a638232a10ec\" (UID: \"ddc019a8-a65c-4f8a-9d98-a638232a10ec\") "
Nov 02 01:51:01.840113 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:01.840079    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddc019a8-a65c-4f8a-9d98-a638232a10ec-kube-api-access-xfs9c\" (UniqueName: \"kubernetes.io/projected/ddc019a8-a65c-4f8a-9d98-a638232a10ec-kube-api-access-xfs9c\") pod \"ddc019a8-a65c-4f8a-9d98-a638232a10ec\" (UID: \"ddc019a8-a65c-4f8a-9d98-a638232a10ec\") "
Nov 02 01:51:01.840193 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:01.840077    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56b5b049-867f-4f13-90e5-d2b280044f2d-maas-api-openshift-ca" pod "56b5b049-867f-4f13-90e5-d2b280044f2d" (UID: "56b5b049-867f-4f13-90e5-d2b280044f2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:51:01.840343 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:01.840323    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddc019a8-a65c-4f8a-9d98-a638232a10ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddc019a8-a65c-4f8a-9d98-a638232a10ec-maas-api-openshift-ca\") pod \"ddc019a8-a65c-4f8a-9d98-a638232a10ec\" (UID: \"ddc019a8-a65c-4f8a-9d98-a638232a10ec\") "
Nov 02 01:51:01.840954 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:01.840923    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc019a8-a65c-4f8a-9d98-a638232a10ec-maas-api-openshift-ca" pod "ddc019a8-a65c-4f8a-9d98-a638232a10ec" (UID: "ddc019a8-a65c-4f8a-9d98-a638232a10ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:51:01.841388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:01.841366    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddc019a8-a65c-4f8a-9d98-a638232a10ec-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:01.841487 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:01.841392    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56b5b049-867f-4f13-90e5-d2b280044f2d-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:01.842197 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:01.842175    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56b5b049-867f-4f13-90e5-d2b280044f2d-maas-api-xks-ca" pod "56b5b049-867f-4f13-90e5-d2b280044f2d" (UID: "56b5b049-867f-4f13-90e5-d2b280044f2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:51:01.842347 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:01.842332    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b5b049-867f-4f13-90e5-d2b280044f2d-kube-api-access-ns9qr" pod "56b5b049-867f-4f13-90e5-d2b280044f2d" (UID: "56b5b049-867f-4f13-90e5-d2b280044f2d"). InnerVolumeSpecName "kube-api-access-ns9qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:51:01.842405 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:01.842392    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddc019a8-a65c-4f8a-9d98-a638232a10ec-maas-api-xks-ca" pod "ddc019a8-a65c-4f8a-9d98-a638232a10ec" (UID: "ddc019a8-a65c-4f8a-9d98-a638232a10ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:51:01.842493 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:01.842472    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddc019a8-a65c-4f8a-9d98-a638232a10ec-kube-api-access-xfs9c" pod "ddc019a8-a65c-4f8a-9d98-a638232a10ec" (UID: "ddc019a8-a65c-4f8a-9d98-a638232a10ec"). InnerVolumeSpecName "kube-api-access-xfs9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:51:01.948511 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:01.948426    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56b5b049-867f-4f13-90e5-d2b280044f2d-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:01.948511 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:01.948465    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns9qr\" (UniqueName: \"kubernetes.io/projected/56b5b049-867f-4f13-90e5-d2b280044f2d-kube-api-access-ns9qr\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:01.948511 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:01.948482    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddc019a8-a65c-4f8a-9d98-a638232a10ec-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:01.948511 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:01.948497    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfs9c\" (UniqueName: \"kubernetes.io/projected/ddc019a8-a65c-4f8a-9d98-a638232a10ec-kube-api-access-xfs9c\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:02.397610 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:02.397574    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-59d70a90-d54879985-jv9l7"]
Nov 02 01:51:02.509001 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:02.508960    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd"]
Nov 02 01:51:02.603432 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:02.603402    5639 generic.go:356] "Generic (PLEG): container finished" podID="3f24c71c-bafc-46fa-b9f4-6ee006c40953" containerID="16d9321f8e6b42fcd33c92c0be4a017e049f75d3dd760ac703b1f2326a1cbb0e" exitCode=0
Nov 02 01:51:02.603842 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:02.603475    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31955345-qdbvw" event={"ID":"3f24c71c-bafc-46fa-b9f4-6ee006c40953","Type":"ContainerDied","Data":"16d9321f8e6b42fcd33c92c0be4a017e049f75d3dd760ac703b1f2326a1cbb0e"}
Nov 02 01:51:02.604901 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:02.604884    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd723454-rgtxn"
Nov 02 01:51:02.604952 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:02.604931    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd723454-rgtxn" event={"ID":"ddc019a8-a65c-4f8a-9d98-a638232a10ec","Type":"ContainerDied","Data":"3d45966600b744451e10ad6538334a81417a197d80d308833e77b71f8ede0c86"}
Nov 02 01:51:02.604992 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:02.604977    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d45966600b744451e10ad6538334a81417a197d80d308833e77b71f8ede0c86"
Nov 02 01:51:02.606580 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:02.606559    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2821b375-j8mr8"
Nov 02 01:51:02.606709 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:02.606594    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2821b375-j8mr8" event={"ID":"56b5b049-867f-4f13-90e5-d2b280044f2d","Type":"ContainerDied","Data":"0aa69326df70f21f5dfc69ffca89b205736fd21506935bfa7bf04e8a478849ed"}
Nov 02 01:51:02.606709 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:02.606628    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0aa69326df70f21f5dfc69ffca89b205736fd21506935bfa7bf04e8a478849ed"
Nov 02 01:51:03.797076 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:03.797051    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31955345-qdbvw"
Nov 02 01:51:03.859349 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:03.852307    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f24c71c-bafc-46fa-b9f4-6ee006c40953-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f24c71c-bafc-46fa-b9f4-6ee006c40953-maas-api-openshift-ca\") pod \"3f24c71c-bafc-46fa-b9f4-6ee006c40953\" (UID: \"3f24c71c-bafc-46fa-b9f4-6ee006c40953\") "
Nov 02 01:51:03.859349 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:03.852441    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f24c71c-bafc-46fa-b9f4-6ee006c40953-kube-api-access-hchfm\" (UniqueName: \"kubernetes.io/projected/3f24c71c-bafc-46fa-b9f4-6ee006c40953-kube-api-access-hchfm\") pod \"3f24c71c-bafc-46fa-b9f4-6ee006c40953\" (UID: \"3f24c71c-bafc-46fa-b9f4-6ee006c40953\") "
Nov 02 01:51:03.859349 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:03.852598    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f24c71c-bafc-46fa-b9f4-6ee006c40953-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f24c71c-bafc-46fa-b9f4-6ee006c40953-maas-api-xks-ca\") pod \"3f24c71c-bafc-46fa-b9f4-6ee006c40953\" (UID: \"3f24c71c-bafc-46fa-b9f4-6ee006c40953\") "
Nov 02 01:51:03.892711 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:03.884705    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f24c71c-bafc-46fa-b9f4-6ee006c40953-maas-api-openshift-ca" pod "3f24c71c-bafc-46fa-b9f4-6ee006c40953" (UID: "3f24c71c-bafc-46fa-b9f4-6ee006c40953"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:51:03.892711 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:03.886884    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f24c71c-bafc-46fa-b9f4-6ee006c40953-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:03.907307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:03.898716    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f24c71c-bafc-46fa-b9f4-6ee006c40953-maas-api-xks-ca" pod "3f24c71c-bafc-46fa-b9f4-6ee006c40953" (UID: "3f24c71c-bafc-46fa-b9f4-6ee006c40953"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:51:03.919166 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:03.914080    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f24c71c-bafc-46fa-b9f4-6ee006c40953-kube-api-access-hchfm" pod "3f24c71c-bafc-46fa-b9f4-6ee006c40953" (UID: "3f24c71c-bafc-46fa-b9f4-6ee006c40953"). InnerVolumeSpecName "kube-api-access-hchfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:51:03.998313 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:03.994788    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hchfm\" (UniqueName: \"kubernetes.io/projected/3f24c71c-bafc-46fa-b9f4-6ee006c40953-kube-api-access-hchfm\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:04.020304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:04.019343    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f24c71c-bafc-46fa-b9f4-6ee006c40953-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:04.626480 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:04.626450    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31955345-qdbvw"
Nov 02 01:51:04.626666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:04.626488    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31955345-qdbvw" event={"ID":"3f24c71c-bafc-46fa-b9f4-6ee006c40953","Type":"ContainerDied","Data":"2f293d276534971c726c4744cb948c38ca4b61261ea5bae49f75dd18e6d02484"}
Nov 02 01:51:04.626666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:04.626544    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f293d276534971c726c4744cb948c38ca4b61261ea5bae49f75dd18e6d02484"
Nov 02 01:51:07.398506 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:07.398441    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-59d70a90-d54879985-jv9l7" podUID="060e39cf-da59-4ca7-9e12-50b88e7e4cad" containerName="payload-processing" containerID="cri-o://c22524ed377e3dc969e68ef431eb38e511f4ecc12248fb3a5966127a0797b0b1" gracePeriod=25
Nov 02 01:51:07.510032 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:07.509992    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd" podUID="b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73" containerName="payload-pre-processing" containerID="cri-o://1ab7fc9ea3c81e8ef07d5c8aa70fe273ab739151cff833d3421298c689e3c454" gracePeriod=25
Nov 02 01:51:07.657653 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:07.657079    5639 generic.go:356] "Generic (PLEG): container finished" podID="060e39cf-da59-4ca7-9e12-50b88e7e4cad" containerID="c22524ed377e3dc969e68ef431eb38e511f4ecc12248fb3a5966127a0797b0b1" exitCode=0
Nov 02 01:51:07.657653 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:07.657561    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-59d70a90-d54879985-jv9l7" event={"ID":"060e39cf-da59-4ca7-9e12-50b88e7e4cad","Type":"ContainerDied","Data":"c22524ed377e3dc969e68ef431eb38e511f4ecc12248fb3a5966127a0797b0b1"}
Nov 02 01:51:07.657653 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:07.657608    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-59d70a90-d54879985-jv9l7" event={"ID":"060e39cf-da59-4ca7-9e12-50b88e7e4cad","Type":"ContainerDied","Data":"9abc11a79afe297c5da52d7d851cd705a77eeacba4de0110067f367ee4f1524e"}
Nov 02 01:51:07.657653 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:07.657625    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9abc11a79afe297c5da52d7d851cd705a77eeacba4de0110067f367ee4f1524e"
Nov 02 01:51:07.662093 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:07.662066    5639 generic.go:356] "Generic (PLEG): container finished" podID="b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73" containerID="1ab7fc9ea3c81e8ef07d5c8aa70fe273ab739151cff833d3421298c689e3c454" exitCode=0
Nov 02 01:51:07.662455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:07.662301    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd" event={"ID":"b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73","Type":"ContainerDied","Data":"1ab7fc9ea3c81e8ef07d5c8aa70fe273ab739151cff833d3421298c689e3c454"}
Nov 02 01:51:07.690338 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:07.690273    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-59d70a90-d54879985-jv9l7"
Nov 02 01:51:07.783406 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:07.783185    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/060e39cf-da59-4ca7-9e12-50b88e7e4cad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060e39cf-da59-4ca7-9e12-50b88e7e4cad-plugins-config-volume\") pod \"060e39cf-da59-4ca7-9e12-50b88e7e4cad\" (UID: \"060e39cf-da59-4ca7-9e12-50b88e7e4cad\") "
Nov 02 01:51:07.783644 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:07.783493    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060e39cf-da59-4ca7-9e12-50b88e7e4cad-kube-api-access-dmbxm\" (UniqueName: \"kubernetes.io/projected/060e39cf-da59-4ca7-9e12-50b88e7e4cad-kube-api-access-dmbxm\") pod \"060e39cf-da59-4ca7-9e12-50b88e7e4cad\" (UID: \"060e39cf-da59-4ca7-9e12-50b88e7e4cad\") "
Nov 02 01:51:07.784025 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:07.783996    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060e39cf-da59-4ca7-9e12-50b88e7e4cad-plugins-config-volume" pod "060e39cf-da59-4ca7-9e12-50b88e7e4cad" (UID: "060e39cf-da59-4ca7-9e12-50b88e7e4cad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:51:07.784675 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:07.784648    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060e39cf-da59-4ca7-9e12-50b88e7e4cad-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:07.786158 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:07.786132    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060e39cf-da59-4ca7-9e12-50b88e7e4cad-kube-api-access-dmbxm" pod "060e39cf-da59-4ca7-9e12-50b88e7e4cad" (UID: "060e39cf-da59-4ca7-9e12-50b88e7e4cad"). InnerVolumeSpecName "kube-api-access-dmbxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:51:07.800542 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:07.800514    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd"
Nov 02 01:51:07.886261 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:07.886227    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73-kube-api-access-8nww7\" (UniqueName: \"kubernetes.io/projected/b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73-kube-api-access-8nww7\") pod \"b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73\" (UID: \"b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73\") "
Nov 02 01:51:07.886440 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:07.886303    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73-plugins-config-volume\") pod \"b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73\" (UID: \"b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73\") "
Nov 02 01:51:07.886701 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:07.886671    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73-plugins-config-volume" pod "b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73" (UID: "b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:51:07.887124 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:07.887099    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmbxm\" (UniqueName: \"kubernetes.io/projected/060e39cf-da59-4ca7-9e12-50b88e7e4cad-kube-api-access-dmbxm\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:07.887218 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:07.887128    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:07.888704 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:07.888672    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73-kube-api-access-8nww7" pod "b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73" (UID: "b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73"). InnerVolumeSpecName "kube-api-access-8nww7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:51:08.170309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:08.158506    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nww7\" (UniqueName: \"kubernetes.io/projected/b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73-kube-api-access-8nww7\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:08.672862 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:08.672827    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd"
Nov 02 01:51:08.673338 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:08.672855    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd" event={"ID":"b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73","Type":"ContainerDied","Data":"64fa45da471fbab24b380e965e9a493b612bb79801bd34df36fc02a4d57483e6"}
Nov 02 01:51:08.673338 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:08.672946    5639 scope.go:122] "RemoveContainer" containerID="1ab7fc9ea3c81e8ef07d5c8aa70fe273ab739151cff833d3421298c689e3c454"
Nov 02 01:51:08.673338 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:08.672989    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-59d70a90-d54879985-jv9l7"
Nov 02 01:51:08.892799 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:08.892768    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd"]
Nov 02 01:51:08.898101 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:08.898079    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-59d70a90-5447856f5-7qfxd"]
Nov 02 01:51:09.005604 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:09.005530    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-59d70a90-d54879985-jv9l7"]
Nov 02 01:51:09.011087 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:09.011064    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-59d70a90-d54879985-jv9l7"]
Nov 02 01:51:09.221052 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:09.221019    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="060e39cf-da59-4ca7-9e12-50b88e7e4cad" path="/var/lib/kubelet/pods/060e39cf-da59-4ca7-9e12-50b88e7e4cad/volumes"
Nov 02 01:51:09.221722 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:09.221688    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73" path="/var/lib/kubelet/pods/b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73/volumes"
Nov 02 01:51:10.382330 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:10.382247    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2"]
Nov 02 01:51:10.506956 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:10.506900    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv"]
Nov 02 01:51:12.824464 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:12.824428    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp"]
Nov 02 01:51:12.949077 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:12.949044    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr"]
Nov 02 01:51:14.182523 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.182491    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fc56410-99dhc"]
Nov 02 01:51:14.183294 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.183263    5639 cpu_manager.go:414] "containerMap: removing container" podUID="3f24c71c-bafc-46fa-b9f4-6ee006c40953" containerName="revoke-keys"
Nov 02 01:51:14.183371 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.183295    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f24c71c-bafc-46fa-b9f4-6ee006c40953" containerName="revoke-keys"
Nov 02 01:51:14.183371 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.183326    5639 cpu_manager.go:414] "containerMap: removing container" podUID="56b5b049-867f-4f13-90e5-d2b280044f2d" containerName="revoke-keys"
Nov 02 01:51:14.183371 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.183332    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56b5b049-867f-4f13-90e5-d2b280044f2d" containerName="revoke-keys"
Nov 02 01:51:14.183371 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.183344    5639 cpu_manager.go:414] "containerMap: removing container" podUID="060e39cf-da59-4ca7-9e12-50b88e7e4cad" containerName="payload-processing"
Nov 02 01:51:14.183371 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.183349    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="060e39cf-da59-4ca7-9e12-50b88e7e4cad" containerName="payload-processing"
Nov 02 01:51:14.183371 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.183360    5639 cpu_manager.go:414] "containerMap: removing container" podUID="ddc019a8-a65c-4f8a-9d98-a638232a10ec" containerName="revoke-keys"
Nov 02 01:51:14.183371 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.183364    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddc019a8-a65c-4f8a-9d98-a638232a10ec" containerName="revoke-keys"
Nov 02 01:51:14.183583 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.183375    5639 cpu_manager.go:414] "containerMap: removing container" podUID="b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73" containerName="payload-pre-processing"
Nov 02 01:51:14.183583 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.183379    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73" containerName="payload-pre-processing"
Nov 02 01:51:14.183583 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.183514    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="060e39cf-da59-4ca7-9e12-50b88e7e4cad" containerName="payload-processing"
Nov 02 01:51:14.183583 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.183524    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddc019a8-a65c-4f8a-9d98-a638232a10ec" containerName="revoke-keys"
Nov 02 01:51:14.183583 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.183531    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2ede435-f1d8-4c62-b0df-d7a2c9e7cb73" containerName="payload-pre-processing"
Nov 02 01:51:14.183583 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.183544    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f24c71c-bafc-46fa-b9f4-6ee006c40953" containerName="revoke-keys"
Nov 02 01:51:14.183583 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.183553    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="56b5b049-867f-4f13-90e5-d2b280044f2d" containerName="revoke-keys"
Nov 02 01:51:14.186934 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.186916    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fc56410-99dhc"
Nov 02 01:51:14.239479 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.239447    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:51:14.239953 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.239544    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:51:14.239953 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.239452    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:51:14.239953 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.239603    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:51:14.244034 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.244012    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fc56410-99dhc"]
Nov 02 01:51:14.331308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.331111    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/514e63d2-f45e-4e85-aa53-1799f73f47ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5fc56410-99dhc\" (UID: \"514e63d2-f45e-4e85-aa53-1799f73f47ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fc56410-99dhc"
Nov 02 01:51:14.331308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.331230    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7fx8\" (UniqueName: \"kubernetes.io/projected/514e63d2-f45e-4e85-aa53-1799f73f47ab-kube-api-access-g7fx8\") pod \"maas-api-revoke-keys-e2e-ait-5fc56410-99dhc\" (UID: \"514e63d2-f45e-4e85-aa53-1799f73f47ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fc56410-99dhc"
Nov 02 01:51:14.331569 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.331419    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/514e63d2-f45e-4e85-aa53-1799f73f47ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5fc56410-99dhc\" (UID: \"514e63d2-f45e-4e85-aa53-1799f73f47ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fc56410-99dhc"
Nov 02 01:51:14.444390 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.440555    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7fx8\" (UniqueName: \"kubernetes.io/projected/514e63d2-f45e-4e85-aa53-1799f73f47ab-kube-api-access-g7fx8\") pod \"maas-api-revoke-keys-e2e-ait-5fc56410-99dhc\" (UID: \"514e63d2-f45e-4e85-aa53-1799f73f47ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fc56410-99dhc"
Nov 02 01:51:14.444390 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.440807    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/514e63d2-f45e-4e85-aa53-1799f73f47ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5fc56410-99dhc\" (UID: \"514e63d2-f45e-4e85-aa53-1799f73f47ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fc56410-99dhc"
Nov 02 01:51:14.444390 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.440984    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/514e63d2-f45e-4e85-aa53-1799f73f47ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5fc56410-99dhc\" (UID: \"514e63d2-f45e-4e85-aa53-1799f73f47ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fc56410-99dhc"
Nov 02 01:51:14.444390 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.441892    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/514e63d2-f45e-4e85-aa53-1799f73f47ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5fc56410-99dhc\" (UID: \"514e63d2-f45e-4e85-aa53-1799f73f47ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fc56410-99dhc"
Nov 02 01:51:14.462314 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.459793    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/514e63d2-f45e-4e85-aa53-1799f73f47ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5fc56410-99dhc\" (UID: \"514e63d2-f45e-4e85-aa53-1799f73f47ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fc56410-99dhc"
Nov 02 01:51:14.511516 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.511482    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7fx8\" (UniqueName: \"kubernetes.io/projected/514e63d2-f45e-4e85-aa53-1799f73f47ab-kube-api-access-g7fx8\") pod \"maas-api-revoke-keys-e2e-ait-5fc56410-99dhc\" (UID: \"514e63d2-f45e-4e85-aa53-1799f73f47ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fc56410-99dhc"
Nov 02 01:51:14.804708 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:14.804616    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fc56410-99dhc"
Nov 02 01:51:15.253036 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:51:15.253004    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514e63d2_f45e_4e85_aa53_1799f73f47ab.slice/crio-de1abdc0c6555eba4ef991cfb03dc7e02ca203391403e789dca777884c67756a WatchSource:0}: Error finding container de1abdc0c6555eba4ef991cfb03dc7e02ca203391403e789dca777884c67756a: Status 404 returned error can't find the container with id de1abdc0c6555eba4ef991cfb03dc7e02ca203391403e789dca777884c67756a
Nov 02 01:51:15.254376 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.254346    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fc56410-99dhc"]
Nov 02 01:51:15.382809 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.382734    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2" podUID="ccbdec1a-9065-4b40-bf67-21d4f8c30e33" containerName="payload-processing" containerID="cri-o://0fa86b5ba2b5c05d41d96be686b42796ccf9cf670c558b66457fe555b4fd104d" gracePeriod=25
Nov 02 01:51:15.507689 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.507640    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv" podUID="ef2ff06b-7835-4075-9346-7dc5ced29cf4" containerName="payload-pre-processing" containerID="cri-o://2b3fb118a3ab70f10756429b597e7c9627e94cdc1c0ce8b36c74b79163a7ee67" gracePeriod=25
Nov 02 01:51:15.676824 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.676453    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2"
Nov 02 01:51:15.775472 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.774791    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccbdec1a-9065-4b40-bf67-21d4f8c30e33-kube-api-access-7t7db\" (UniqueName: \"kubernetes.io/projected/ccbdec1a-9065-4b40-bf67-21d4f8c30e33-kube-api-access-7t7db\") pod \"ccbdec1a-9065-4b40-bf67-21d4f8c30e33\" (UID: \"ccbdec1a-9065-4b40-bf67-21d4f8c30e33\") "
Nov 02 01:51:15.775472 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.775071    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccbdec1a-9065-4b40-bf67-21d4f8c30e33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccbdec1a-9065-4b40-bf67-21d4f8c30e33-plugins-config-volume\") pod \"ccbdec1a-9065-4b40-bf67-21d4f8c30e33\" (UID: \"ccbdec1a-9065-4b40-bf67-21d4f8c30e33\") "
Nov 02 01:51:15.782979 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.778137    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccbdec1a-9065-4b40-bf67-21d4f8c30e33-kube-api-access-7t7db" pod "ccbdec1a-9065-4b40-bf67-21d4f8c30e33" (UID: "ccbdec1a-9065-4b40-bf67-21d4f8c30e33"). InnerVolumeSpecName "kube-api-access-7t7db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:51:15.794837 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.794805    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t7db\" (UniqueName: \"kubernetes.io/projected/ccbdec1a-9065-4b40-bf67-21d4f8c30e33-kube-api-access-7t7db\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:15.797661 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.797631    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fc56410-99dhc" event={"ID":"514e63d2-f45e-4e85-aa53-1799f73f47ab","Type":"ContainerStarted","Data":"2999e7ee14e40487b8464e2827e4cfc186bfaf4ba38e0a5f7c882819ac5a667c"}
Nov 02 01:51:15.797796 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.797732    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fc56410-99dhc" event={"ID":"514e63d2-f45e-4e85-aa53-1799f73f47ab","Type":"ContainerStarted","Data":"de1abdc0c6555eba4ef991cfb03dc7e02ca203391403e789dca777884c67756a"}
Nov 02 01:51:15.798152 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.798127    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv"
Nov 02 01:51:15.799605 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.799575    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccbdec1a-9065-4b40-bf67-21d4f8c30e33-plugins-config-volume" pod "ccbdec1a-9065-4b40-bf67-21d4f8c30e33" (UID: "ccbdec1a-9065-4b40-bf67-21d4f8c30e33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:51:15.800874 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.800851    5639 generic.go:356] "Generic (PLEG): container finished" podID="ef2ff06b-7835-4075-9346-7dc5ced29cf4" containerID="2b3fb118a3ab70f10756429b597e7c9627e94cdc1c0ce8b36c74b79163a7ee67" exitCode=0
Nov 02 01:51:15.800978 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.800920    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv" event={"ID":"ef2ff06b-7835-4075-9346-7dc5ced29cf4","Type":"ContainerDied","Data":"2b3fb118a3ab70f10756429b597e7c9627e94cdc1c0ce8b36c74b79163a7ee67"}
Nov 02 01:51:15.800978 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.800970    5639 scope.go:122] "RemoveContainer" containerID="2b3fb118a3ab70f10756429b597e7c9627e94cdc1c0ce8b36c74b79163a7ee67"
Nov 02 01:51:15.803635 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.803610    5639 generic.go:356] "Generic (PLEG): container finished" podID="ccbdec1a-9065-4b40-bf67-21d4f8c30e33" containerID="0fa86b5ba2b5c05d41d96be686b42796ccf9cf670c558b66457fe555b4fd104d" exitCode=0
Nov 02 01:51:15.803743 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.803733    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2"
Nov 02 01:51:15.803863 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.803844    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2" event={"ID":"ccbdec1a-9065-4b40-bf67-21d4f8c30e33","Type":"ContainerDied","Data":"0fa86b5ba2b5c05d41d96be686b42796ccf9cf670c558b66457fe555b4fd104d"}
Nov 02 01:51:15.803920 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.803884    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2" event={"ID":"ccbdec1a-9065-4b40-bf67-21d4f8c30e33","Type":"ContainerDied","Data":"9fd88608f3d4560166ff80e6bb53eebc7f737837814d50fbb9402c016c3e1ac6"}
Nov 02 01:51:15.814612 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.814317    5639 scope.go:122] "RemoveContainer" containerID="0fa86b5ba2b5c05d41d96be686b42796ccf9cf670c558b66457fe555b4fd104d"
Nov 02 01:51:15.824695 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.824674    5639 scope.go:122] "RemoveContainer" containerID="0fa86b5ba2b5c05d41d96be686b42796ccf9cf670c558b66457fe555b4fd104d"
Nov 02 01:51:15.824970 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:51:15.824950    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fa86b5ba2b5c05d41d96be686b42796ccf9cf670c558b66457fe555b4fd104d\": container with ID starting with 0fa86b5ba2b5c05d41d96be686b42796ccf9cf670c558b66457fe555b4fd104d not found: ID does not exist" containerID="0fa86b5ba2b5c05d41d96be686b42796ccf9cf670c558b66457fe555b4fd104d"
Nov 02 01:51:15.825015 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.824980    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fa86b5ba2b5c05d41d96be686b42796ccf9cf670c558b66457fe555b4fd104d"} err="failed to get container status \"0fa86b5ba2b5c05d41d96be686b42796ccf9cf670c558b66457fe555b4fd104d\": rpc error: code = NotFound desc = could not find container \"0fa86b5ba2b5c05d41d96be686b42796ccf9cf670c558b66457fe555b4fd104d\": container with ID starting with 0fa86b5ba2b5c05d41d96be686b42796ccf9cf670c558b66457fe555b4fd104d not found: ID does not exist"
Nov 02 01:51:15.896319 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.896273    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef2ff06b-7835-4075-9346-7dc5ced29cf4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2ff06b-7835-4075-9346-7dc5ced29cf4-plugins-config-volume\") pod \"ef2ff06b-7835-4075-9346-7dc5ced29cf4\" (UID: \"ef2ff06b-7835-4075-9346-7dc5ced29cf4\") "
Nov 02 01:51:15.896631 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.896617    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef2ff06b-7835-4075-9346-7dc5ced29cf4-kube-api-access-2r85r\" (UniqueName: \"kubernetes.io/projected/ef2ff06b-7835-4075-9346-7dc5ced29cf4-kube-api-access-2r85r\") pod \"ef2ff06b-7835-4075-9346-7dc5ced29cf4\" (UID: \"ef2ff06b-7835-4075-9346-7dc5ced29cf4\") "
Nov 02 01:51:15.896739 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.896673    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef2ff06b-7835-4075-9346-7dc5ced29cf4-plugins-config-volume" pod "ef2ff06b-7835-4075-9346-7dc5ced29cf4" (UID: "ef2ff06b-7835-4075-9346-7dc5ced29cf4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:51:15.897256 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.897238    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2ff06b-7835-4075-9346-7dc5ced29cf4-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:15.897256 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.897256    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccbdec1a-9065-4b40-bf67-21d4f8c30e33-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:15.899008 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.898985    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef2ff06b-7835-4075-9346-7dc5ced29cf4-kube-api-access-2r85r" pod "ef2ff06b-7835-4075-9346-7dc5ced29cf4" (UID: "ef2ff06b-7835-4075-9346-7dc5ced29cf4"). InnerVolumeSpecName "kube-api-access-2r85r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:51:15.999591 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:15.999554    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r85r\" (UniqueName: \"kubernetes.io/projected/ef2ff06b-7835-4075-9346-7dc5ced29cf4-kube-api-access-2r85r\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:16.075356 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:16.071435    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fc56410-99dhc" podStartSLOduration=2.0714154 podStartE2EDuration="2.0714154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:51:16.068032673 +0000 UTC m=+1540.971714789" watchObservedRunningTime="2026-11-02 01:51:16.0714154 +0000 UTC m=+1540.975097515"
Nov 02 01:51:16.183051 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:16.183020    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2"]
Nov 02 01:51:16.188980 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:16.188953    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fd723454-6bc6b6856f-cfgh2"]
Nov 02 01:51:16.816354 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:16.816216    5639 generic.go:356] "Generic (PLEG): container finished" podID="514e63d2-f45e-4e85-aa53-1799f73f47ab" containerID="2999e7ee14e40487b8464e2827e4cfc186bfaf4ba38e0a5f7c882819ac5a667c" exitCode=0
Nov 02 01:51:16.816354 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:16.816328    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fc56410-99dhc" event={"ID":"514e63d2-f45e-4e85-aa53-1799f73f47ab","Type":"ContainerDied","Data":"2999e7ee14e40487b8464e2827e4cfc186bfaf4ba38e0a5f7c882819ac5a667c"}
Nov 02 01:51:16.817636 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:16.817614    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv"
Nov 02 01:51:16.817755 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:16.817663    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv" event={"ID":"ef2ff06b-7835-4075-9346-7dc5ced29cf4","Type":"ContainerDied","Data":"54776c43e49060a976bd6015f99a7778385393014e8a37a452d16f4e1fe628f8"}
Nov 02 01:51:17.048342 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:17.048306    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv"]
Nov 02 01:51:17.053395 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:17.053374    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fd723454-68f7c8d6f9-gvgzv"]
Nov 02 01:51:17.220861 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:17.220830    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccbdec1a-9065-4b40-bf67-21d4f8c30e33" path="/var/lib/kubelet/pods/ccbdec1a-9065-4b40-bf67-21d4f8c30e33/volumes"
Nov 02 01:51:17.221348 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:17.221325    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef2ff06b-7835-4075-9346-7dc5ced29cf4" path="/var/lib/kubelet/pods/ef2ff06b-7835-4075-9346-7dc5ced29cf4/volumes"
Nov 02 01:51:17.824811 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:17.824771    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp" podUID="79481b1b-92f4-4dc0-8ac1-b03e509606c3" containerName="payload-processing" containerID="cri-o://721675fa48a2f22b0432bad608db84fbbb1cd85c6e34ac67fa25b2284b5c649c" gracePeriod=25
Nov 02 01:51:17.949805 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:17.949773    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr" podUID="405c8e1f-2f4b-4624-ac6c-5f607e4bb81d" containerName="payload-pre-processing" containerID="cri-o://6f5bab36ea51048cec6ac63c4395edea8d7de93638234c03d77d93457eb0a134" gracePeriod=25
Nov 02 01:51:18.108615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.108589    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fc56410-99dhc"
Nov 02 01:51:18.116873 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.116848    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp"
Nov 02 01:51:18.218687 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.218662    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr"
Nov 02 01:51:18.236416 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.236379    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79481b1b-92f4-4dc0-8ac1-b03e509606c3-kube-api-access-tlrc7\" (UniqueName: \"kubernetes.io/projected/79481b1b-92f4-4dc0-8ac1-b03e509606c3-kube-api-access-tlrc7\") pod \"79481b1b-92f4-4dc0-8ac1-b03e509606c3\" (UID: \"79481b1b-92f4-4dc0-8ac1-b03e509606c3\") "
Nov 02 01:51:18.236571 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.236457    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/514e63d2-f45e-4e85-aa53-1799f73f47ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/514e63d2-f45e-4e85-aa53-1799f73f47ab-maas-api-xks-ca\") pod \"514e63d2-f45e-4e85-aa53-1799f73f47ab\" (UID: \"514e63d2-f45e-4e85-aa53-1799f73f47ab\") "
Nov 02 01:51:18.236571 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.236516    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/514e63d2-f45e-4e85-aa53-1799f73f47ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/514e63d2-f45e-4e85-aa53-1799f73f47ab-maas-api-openshift-ca\") pod \"514e63d2-f45e-4e85-aa53-1799f73f47ab\" (UID: \"514e63d2-f45e-4e85-aa53-1799f73f47ab\") "
Nov 02 01:51:18.236694 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.236597    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/514e63d2-f45e-4e85-aa53-1799f73f47ab-kube-api-access-g7fx8\" (UniqueName: \"kubernetes.io/projected/514e63d2-f45e-4e85-aa53-1799f73f47ab-kube-api-access-g7fx8\") pod \"514e63d2-f45e-4e85-aa53-1799f73f47ab\" (UID: \"514e63d2-f45e-4e85-aa53-1799f73f47ab\") "
Nov 02 01:51:18.236768 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.236749    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79481b1b-92f4-4dc0-8ac1-b03e509606c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79481b1b-92f4-4dc0-8ac1-b03e509606c3-plugins-config-volume\") pod \"79481b1b-92f4-4dc0-8ac1-b03e509606c3\" (UID: \"79481b1b-92f4-4dc0-8ac1-b03e509606c3\") "
Nov 02 01:51:18.236992 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.236962    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514e63d2-f45e-4e85-aa53-1799f73f47ab-maas-api-openshift-ca" pod "514e63d2-f45e-4e85-aa53-1799f73f47ab" (UID: "514e63d2-f45e-4e85-aa53-1799f73f47ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:51:18.237361 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.237335    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79481b1b-92f4-4dc0-8ac1-b03e509606c3-plugins-config-volume" pod "79481b1b-92f4-4dc0-8ac1-b03e509606c3" (UID: "79481b1b-92f4-4dc0-8ac1-b03e509606c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:51:18.237669 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.237647    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/514e63d2-f45e-4e85-aa53-1799f73f47ab-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:18.237752 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.237673    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79481b1b-92f4-4dc0-8ac1-b03e509606c3-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:18.238895 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.238870    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/514e63d2-f45e-4e85-aa53-1799f73f47ab-maas-api-xks-ca" pod "514e63d2-f45e-4e85-aa53-1799f73f47ab" (UID: "514e63d2-f45e-4e85-aa53-1799f73f47ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:51:18.239609 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.239586    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/514e63d2-f45e-4e85-aa53-1799f73f47ab-kube-api-access-g7fx8" pod "514e63d2-f45e-4e85-aa53-1799f73f47ab" (UID: "514e63d2-f45e-4e85-aa53-1799f73f47ab"). InnerVolumeSpecName "kube-api-access-g7fx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:51:18.239708 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.239661    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79481b1b-92f4-4dc0-8ac1-b03e509606c3-kube-api-access-tlrc7" pod "79481b1b-92f4-4dc0-8ac1-b03e509606c3" (UID: "79481b1b-92f4-4dc0-8ac1-b03e509606c3"). InnerVolumeSpecName "kube-api-access-tlrc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:51:18.339265 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.339234    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405c8e1f-2f4b-4624-ac6c-5f607e4bb81d-kube-api-access-pjfvl\" (UniqueName: \"kubernetes.io/projected/405c8e1f-2f4b-4624-ac6c-5f607e4bb81d-kube-api-access-pjfvl\") pod \"405c8e1f-2f4b-4624-ac6c-5f607e4bb81d\" (UID: \"405c8e1f-2f4b-4624-ac6c-5f607e4bb81d\") "
Nov 02 01:51:18.339446 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.339311    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/405c8e1f-2f4b-4624-ac6c-5f607e4bb81d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405c8e1f-2f4b-4624-ac6c-5f607e4bb81d-plugins-config-volume\") pod \"405c8e1f-2f4b-4624-ac6c-5f607e4bb81d\" (UID: \"405c8e1f-2f4b-4624-ac6c-5f607e4bb81d\") "
Nov 02 01:51:18.339702 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.339680    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405c8e1f-2f4b-4624-ac6c-5f607e4bb81d-plugins-config-volume" pod "405c8e1f-2f4b-4624-ac6c-5f607e4bb81d" (UID: "405c8e1f-2f4b-4624-ac6c-5f607e4bb81d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:51:18.340320 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.340260    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405c8e1f-2f4b-4624-ac6c-5f607e4bb81d-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:18.340451 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.340351    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlrc7\" (UniqueName: \"kubernetes.io/projected/79481b1b-92f4-4dc0-8ac1-b03e509606c3-kube-api-access-tlrc7\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:18.340451 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.340365    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/514e63d2-f45e-4e85-aa53-1799f73f47ab-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:18.340451 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.340374    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7fx8\" (UniqueName: \"kubernetes.io/projected/514e63d2-f45e-4e85-aa53-1799f73f47ab-kube-api-access-g7fx8\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:18.342230 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.342190    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405c8e1f-2f4b-4624-ac6c-5f607e4bb81d-kube-api-access-pjfvl" pod "405c8e1f-2f4b-4624-ac6c-5f607e4bb81d" (UID: "405c8e1f-2f4b-4624-ac6c-5f607e4bb81d"). InnerVolumeSpecName "kube-api-access-pjfvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:51:18.442610 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.442532    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjfvl\" (UniqueName: \"kubernetes.io/projected/405c8e1f-2f4b-4624-ac6c-5f607e4bb81d-kube-api-access-pjfvl\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:18.853502 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.853471    5639 generic.go:356] "Generic (PLEG): container finished" podID="79481b1b-92f4-4dc0-8ac1-b03e509606c3" containerID="721675fa48a2f22b0432bad608db84fbbb1cd85c6e34ac67fa25b2284b5c649c" exitCode=0
Nov 02 01:51:18.854008 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.853548    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp"
Nov 02 01:51:18.854008 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.853560    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp" event={"ID":"79481b1b-92f4-4dc0-8ac1-b03e509606c3","Type":"ContainerDied","Data":"721675fa48a2f22b0432bad608db84fbbb1cd85c6e34ac67fa25b2284b5c649c"}
Nov 02 01:51:18.854008 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.853747    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp" event={"ID":"79481b1b-92f4-4dc0-8ac1-b03e509606c3","Type":"ContainerDied","Data":"9c90434981661a4d9b9f395e0299ee6f798ac625ded1c90ac6b9b7738d71876f"}
Nov 02 01:51:18.854008 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.853788    5639 scope.go:122] "RemoveContainer" containerID="721675fa48a2f22b0432bad608db84fbbb1cd85c6e34ac67fa25b2284b5c649c"
Nov 02 01:51:18.855577 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.855428    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fc56410-99dhc" event={"ID":"514e63d2-f45e-4e85-aa53-1799f73f47ab","Type":"ContainerDied","Data":"de1abdc0c6555eba4ef991cfb03dc7e02ca203391403e789dca777884c67756a"}
Nov 02 01:51:18.855577 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.855437    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fc56410-99dhc"
Nov 02 01:51:18.855577 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.855470    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de1abdc0c6555eba4ef991cfb03dc7e02ca203391403e789dca777884c67756a"
Nov 02 01:51:18.857223 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.857202    5639 generic.go:356] "Generic (PLEG): container finished" podID="405c8e1f-2f4b-4624-ac6c-5f607e4bb81d" containerID="6f5bab36ea51048cec6ac63c4395edea8d7de93638234c03d77d93457eb0a134" exitCode=0
Nov 02 01:51:18.857498 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.857477    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr" event={"ID":"405c8e1f-2f4b-4624-ac6c-5f607e4bb81d","Type":"ContainerDied","Data":"6f5bab36ea51048cec6ac63c4395edea8d7de93638234c03d77d93457eb0a134"}
Nov 02 01:51:18.857609 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.857526    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr" event={"ID":"405c8e1f-2f4b-4624-ac6c-5f607e4bb81d","Type":"ContainerDied","Data":"a0e2088bec68963f2e08fb82be8ff0c8408db90a080a51967efcff7156d2cfa5"}
Nov 02 01:51:18.857672 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.857611    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr"
Nov 02 01:51:18.869879 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.869861    5639 scope.go:122] "RemoveContainer" containerID="721675fa48a2f22b0432bad608db84fbbb1cd85c6e34ac67fa25b2284b5c649c"
Nov 02 01:51:18.870219 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:51:18.870192    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"721675fa48a2f22b0432bad608db84fbbb1cd85c6e34ac67fa25b2284b5c649c\": container with ID starting with 721675fa48a2f22b0432bad608db84fbbb1cd85c6e34ac67fa25b2284b5c649c not found: ID does not exist" containerID="721675fa48a2f22b0432bad608db84fbbb1cd85c6e34ac67fa25b2284b5c649c"
Nov 02 01:51:18.870272 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.870231    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"721675fa48a2f22b0432bad608db84fbbb1cd85c6e34ac67fa25b2284b5c649c"} err="failed to get container status \"721675fa48a2f22b0432bad608db84fbbb1cd85c6e34ac67fa25b2284b5c649c\": rpc error: code = NotFound desc = could not find container \"721675fa48a2f22b0432bad608db84fbbb1cd85c6e34ac67fa25b2284b5c649c\": container with ID starting with 721675fa48a2f22b0432bad608db84fbbb1cd85c6e34ac67fa25b2284b5c649c not found: ID does not exist"
Nov 02 01:51:18.870272 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.870255    5639 scope.go:122] "RemoveContainer" containerID="6f5bab36ea51048cec6ac63c4395edea8d7de93638234c03d77d93457eb0a134"
Nov 02 01:51:18.880819 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.880800    5639 scope.go:122] "RemoveContainer" containerID="6f5bab36ea51048cec6ac63c4395edea8d7de93638234c03d77d93457eb0a134"
Nov 02 01:51:18.881091 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:51:18.881071    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f5bab36ea51048cec6ac63c4395edea8d7de93638234c03d77d93457eb0a134\": container with ID starting with 6f5bab36ea51048cec6ac63c4395edea8d7de93638234c03d77d93457eb0a134 not found: ID does not exist" containerID="6f5bab36ea51048cec6ac63c4395edea8d7de93638234c03d77d93457eb0a134"
Nov 02 01:51:18.881156 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:18.881103    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f5bab36ea51048cec6ac63c4395edea8d7de93638234c03d77d93457eb0a134"} err="failed to get container status \"6f5bab36ea51048cec6ac63c4395edea8d7de93638234c03d77d93457eb0a134\": rpc error: code = NotFound desc = could not find container \"6f5bab36ea51048cec6ac63c4395edea8d7de93638234c03d77d93457eb0a134\": container with ID starting with 6f5bab36ea51048cec6ac63c4395edea8d7de93638234c03d77d93457eb0a134 not found: ID does not exist"
Nov 02 01:51:19.021044 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:19.021010    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp"]
Nov 02 01:51:19.070602 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:19.070572    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2821b375-7799bbf8f9-8ptjp"]
Nov 02 01:51:19.132609 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:19.132525    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr"]
Nov 02 01:51:19.137546 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:19.137522    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2821b375-b59976fd7-27bzr"]
Nov 02 01:51:19.221033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:19.220997    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="405c8e1f-2f4b-4624-ac6c-5f607e4bb81d" path="/var/lib/kubelet/pods/405c8e1f-2f4b-4624-ac6c-5f607e4bb81d/volumes"
Nov 02 01:51:19.221521 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:19.221506    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79481b1b-92f4-4dc0-8ac1-b03e509606c3" path="/var/lib/kubelet/pods/79481b1b-92f4-4dc0-8ac1-b03e509606c3/volumes"
Nov 02 01:51:25.688598 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:25.688562    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q"]
Nov 02 01:51:25.810371 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:25.808545    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt"]
Nov 02 01:51:27.474311 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:51:27.467509    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:51:30.946984 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:30.946937    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt" podUID="1a0ddab3-c6be-44e4-800c-012437a54a2e" containerName="payload-pre-processing" containerID="cri-o://ab7c062a6869c0c1efc22b5373be12c6613d4820cb1b9c0e5dfae1118f0c698c" gracePeriod=25
Nov 02 01:51:30.947478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:30.946954    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q" podUID="7392e99a-1aa8-48f9-ab30-4fea2eab88de" containerName="payload-processing" containerID="cri-o://ddbce4175bb3059478807e91aac4e5dff685c9ef7decdfa9f47a86c543348254" gracePeriod=25
Nov 02 01:51:31.258050 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:31.258021    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q"
Nov 02 01:51:31.282248 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:31.282223    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt"
Nov 02 01:51:31.345688 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:31.345650    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7392e99a-1aa8-48f9-ab30-4fea2eab88de-kube-api-access-cx4cb\" (UniqueName: \"kubernetes.io/projected/7392e99a-1aa8-48f9-ab30-4fea2eab88de-kube-api-access-cx4cb\") pod \"7392e99a-1aa8-48f9-ab30-4fea2eab88de\" (UID: \"7392e99a-1aa8-48f9-ab30-4fea2eab88de\") "
Nov 02 01:51:31.345863 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:31.345748    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a0ddab3-c6be-44e4-800c-012437a54a2e-kube-api-access-vrrf8\" (UniqueName: \"kubernetes.io/projected/1a0ddab3-c6be-44e4-800c-012437a54a2e-kube-api-access-vrrf8\") pod \"1a0ddab3-c6be-44e4-800c-012437a54a2e\" (UID: \"1a0ddab3-c6be-44e4-800c-012437a54a2e\") "
Nov 02 01:51:31.345934 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:31.345919    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7392e99a-1aa8-48f9-ab30-4fea2eab88de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7392e99a-1aa8-48f9-ab30-4fea2eab88de-plugins-config-volume\") pod \"7392e99a-1aa8-48f9-ab30-4fea2eab88de\" (UID: \"7392e99a-1aa8-48f9-ab30-4fea2eab88de\") "
Nov 02 01:51:31.346014 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:31.346001    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a0ddab3-c6be-44e4-800c-012437a54a2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a0ddab3-c6be-44e4-800c-012437a54a2e-plugins-config-volume\") pod \"1a0ddab3-c6be-44e4-800c-012437a54a2e\" (UID: \"1a0ddab3-c6be-44e4-800c-012437a54a2e\") "
Nov 02 01:51:31.346328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:31.346302    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7392e99a-1aa8-48f9-ab30-4fea2eab88de-plugins-config-volume" pod "7392e99a-1aa8-48f9-ab30-4fea2eab88de" (UID: "7392e99a-1aa8-48f9-ab30-4fea2eab88de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:51:31.346421 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:31.346365    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a0ddab3-c6be-44e4-800c-012437a54a2e-plugins-config-volume" pod "1a0ddab3-c6be-44e4-800c-012437a54a2e" (UID: "1a0ddab3-c6be-44e4-800c-012437a54a2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:51:31.346682 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:31.346665    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7392e99a-1aa8-48f9-ab30-4fea2eab88de-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:31.346733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:31.346685    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a0ddab3-c6be-44e4-800c-012437a54a2e-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:31.348559 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:31.348533    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7392e99a-1aa8-48f9-ab30-4fea2eab88de-kube-api-access-cx4cb" pod "7392e99a-1aa8-48f9-ab30-4fea2eab88de" (UID: "7392e99a-1aa8-48f9-ab30-4fea2eab88de"). InnerVolumeSpecName "kube-api-access-cx4cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:51:31.348640 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:31.348565    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a0ddab3-c6be-44e4-800c-012437a54a2e-kube-api-access-vrrf8" pod "1a0ddab3-c6be-44e4-800c-012437a54a2e" (UID: "1a0ddab3-c6be-44e4-800c-012437a54a2e"). InnerVolumeSpecName "kube-api-access-vrrf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:51:31.448207 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:31.448174    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrrf8\" (UniqueName: \"kubernetes.io/projected/1a0ddab3-c6be-44e4-800c-012437a54a2e-kube-api-access-vrrf8\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:31.448207 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:31.448205    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx4cb\" (UniqueName: \"kubernetes.io/projected/7392e99a-1aa8-48f9-ab30-4fea2eab88de-kube-api-access-cx4cb\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:32.073610 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:32.073577    5639 generic.go:356] "Generic (PLEG): container finished" podID="7392e99a-1aa8-48f9-ab30-4fea2eab88de" containerID="ddbce4175bb3059478807e91aac4e5dff685c9ef7decdfa9f47a86c543348254" exitCode=0
Nov 02 01:51:32.074047 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:32.073640    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q"
Nov 02 01:51:32.074047 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:32.073657    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q" event={"ID":"7392e99a-1aa8-48f9-ab30-4fea2eab88de","Type":"ContainerDied","Data":"ddbce4175bb3059478807e91aac4e5dff685c9ef7decdfa9f47a86c543348254"}
Nov 02 01:51:32.074047 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:32.073742    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q" event={"ID":"7392e99a-1aa8-48f9-ab30-4fea2eab88de","Type":"ContainerDied","Data":"38edba4917ee3ed4decc038df2172e853ac02a7e9087213a10a953dbf8a0897c"}
Nov 02 01:51:32.074047 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:32.073774    5639 scope.go:122] "RemoveContainer" containerID="ddbce4175bb3059478807e91aac4e5dff685c9ef7decdfa9f47a86c543348254"
Nov 02 01:51:32.075629 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:32.075604    5639 generic.go:356] "Generic (PLEG): container finished" podID="1a0ddab3-c6be-44e4-800c-012437a54a2e" containerID="ab7c062a6869c0c1efc22b5373be12c6613d4820cb1b9c0e5dfae1118f0c698c" exitCode=0
Nov 02 01:51:32.075766 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:32.075660    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt"
Nov 02 01:51:32.075766 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:32.075666    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt" event={"ID":"1a0ddab3-c6be-44e4-800c-012437a54a2e","Type":"ContainerDied","Data":"ab7c062a6869c0c1efc22b5373be12c6613d4820cb1b9c0e5dfae1118f0c698c"}
Nov 02 01:51:32.075766 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:32.075704    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt" event={"ID":"1a0ddab3-c6be-44e4-800c-012437a54a2e","Type":"ContainerDied","Data":"592066d617951d2c8c6679033715c7c69c64b8a719b0b0f982b601ba5052e2cd"}
Nov 02 01:51:32.086168 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:32.086154    5639 scope.go:122] "RemoveContainer" containerID="ddbce4175bb3059478807e91aac4e5dff685c9ef7decdfa9f47a86c543348254"
Nov 02 01:51:32.086423 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:51:32.086401    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddbce4175bb3059478807e91aac4e5dff685c9ef7decdfa9f47a86c543348254\": container with ID starting with ddbce4175bb3059478807e91aac4e5dff685c9ef7decdfa9f47a86c543348254 not found: ID does not exist" containerID="ddbce4175bb3059478807e91aac4e5dff685c9ef7decdfa9f47a86c543348254"
Nov 02 01:51:32.086471 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:32.086431    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddbce4175bb3059478807e91aac4e5dff685c9ef7decdfa9f47a86c543348254"} err="failed to get container status \"ddbce4175bb3059478807e91aac4e5dff685c9ef7decdfa9f47a86c543348254\": rpc error: code = NotFound desc = could not find container \"ddbce4175bb3059478807e91aac4e5dff685c9ef7decdfa9f47a86c543348254\": container with ID starting with ddbce4175bb3059478807e91aac4e5dff685c9ef7decdfa9f47a86c543348254 not found: ID does not exist"
Nov 02 01:51:32.086471 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:32.086449    5639 scope.go:122] "RemoveContainer" containerID="ab7c062a6869c0c1efc22b5373be12c6613d4820cb1b9c0e5dfae1118f0c698c"
Nov 02 01:51:32.097239 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:32.097224    5639 scope.go:122] "RemoveContainer" containerID="ab7c062a6869c0c1efc22b5373be12c6613d4820cb1b9c0e5dfae1118f0c698c"
Nov 02 01:51:32.097572 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:51:32.097550    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab7c062a6869c0c1efc22b5373be12c6613d4820cb1b9c0e5dfae1118f0c698c\": container with ID starting with ab7c062a6869c0c1efc22b5373be12c6613d4820cb1b9c0e5dfae1118f0c698c not found: ID does not exist" containerID="ab7c062a6869c0c1efc22b5373be12c6613d4820cb1b9c0e5dfae1118f0c698c"
Nov 02 01:51:32.097676 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:32.097587    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7c062a6869c0c1efc22b5373be12c6613d4820cb1b9c0e5dfae1118f0c698c"} err="failed to get container status \"ab7c062a6869c0c1efc22b5373be12c6613d4820cb1b9c0e5dfae1118f0c698c\": rpc error: code = NotFound desc = could not find container \"ab7c062a6869c0c1efc22b5373be12c6613d4820cb1b9c0e5dfae1118f0c698c\": container with ID starting with ab7c062a6869c0c1efc22b5373be12c6613d4820cb1b9c0e5dfae1118f0c698c not found: ID does not exist"
Nov 02 01:51:32.240734 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:32.240657    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt"]
Nov 02 01:51:32.291164 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:32.291136    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5fc56410-54b5cd4867-6t5dt"]
Nov 02 01:51:32.351886 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:32.351851    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q"]
Nov 02 01:51:32.358530 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:32.358503    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5fc56410-6f589674cf-cw67q"]
Nov 02 01:51:33.222201 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:33.222168    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a0ddab3-c6be-44e4-800c-012437a54a2e" path="/var/lib/kubelet/pods/1a0ddab3-c6be-44e4-800c-012437a54a2e/volumes"
Nov 02 01:51:33.222695 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:33.222679    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7392e99a-1aa8-48f9-ab30-4fea2eab88de" path="/var/lib/kubelet/pods/7392e99a-1aa8-48f9-ab30-4fea2eab88de/volumes"
Nov 02 01:51:41.050259 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:41.050209    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffa5e8cd-8654k"]
Nov 02 01:51:41.101096 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:41.101060    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffa5e8cd-8654k"]
Nov 02 01:51:41.220808 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:41.220773    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="febf7f8f-3020-4264-bab0-4e25622b2696" path="/var/lib/kubelet/pods/febf7f8f-3020-4264-bab0-4e25622b2696/volumes"
Nov 02 01:51:45.945272 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:45.945233    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59d70a90-5thbz"]
Nov 02 01:51:45.953051 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:45.953019    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59d70a90-5thbz"]
Nov 02 01:51:47.206328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.203355    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2821b375-j8mr8"]
Nov 02 01:51:47.288104 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.288064    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c0947ba-7328-4226-8fe2-8a781d8c9748" path="/var/lib/kubelet/pods/6c0947ba-7328-4226-8fe2-8a781d8c9748/volumes"
Nov 02 01:51:47.288903 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.288878    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2821b375-j8mr8"]
Nov 02 01:51:47.535609 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.535521    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"]
Nov 02 01:51:47.536418 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.536397    5639 cpu_manager.go:414] "containerMap: removing container" podUID="ef2ff06b-7835-4075-9346-7dc5ced29cf4" containerName="payload-pre-processing"
Nov 02 01:51:47.536418 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.536416    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef2ff06b-7835-4075-9346-7dc5ced29cf4" containerName="payload-pre-processing"
Nov 02 01:51:47.536621 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.536432    5639 cpu_manager.go:414] "containerMap: removing container" podUID="ccbdec1a-9065-4b40-bf67-21d4f8c30e33" containerName="payload-processing"
Nov 02 01:51:47.536621 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.536438    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccbdec1a-9065-4b40-bf67-21d4f8c30e33" containerName="payload-processing"
Nov 02 01:51:47.536621 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.536460    5639 cpu_manager.go:414] "containerMap: removing container" podUID="1a0ddab3-c6be-44e4-800c-012437a54a2e" containerName="payload-pre-processing"
Nov 02 01:51:47.536621 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.536465    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a0ddab3-c6be-44e4-800c-012437a54a2e" containerName="payload-pre-processing"
Nov 02 01:51:47.536621 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.536491    5639 cpu_manager.go:414] "containerMap: removing container" podUID="405c8e1f-2f4b-4624-ac6c-5f607e4bb81d" containerName="payload-pre-processing"
Nov 02 01:51:47.536621 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.536497    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405c8e1f-2f4b-4624-ac6c-5f607e4bb81d" containerName="payload-pre-processing"
Nov 02 01:51:47.536621 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.536510    5639 cpu_manager.go:414] "containerMap: removing container" podUID="514e63d2-f45e-4e85-aa53-1799f73f47ab" containerName="revoke-keys"
Nov 02 01:51:47.536621 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.536517    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="514e63d2-f45e-4e85-aa53-1799f73f47ab" containerName="revoke-keys"
Nov 02 01:51:47.536621 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.536541    5639 cpu_manager.go:414] "containerMap: removing container" podUID="7392e99a-1aa8-48f9-ab30-4fea2eab88de" containerName="payload-processing"
Nov 02 01:51:47.536621 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.536548    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7392e99a-1aa8-48f9-ab30-4fea2eab88de" containerName="payload-processing"
Nov 02 01:51:47.536621 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.536575    5639 cpu_manager.go:414] "containerMap: removing container" podUID="79481b1b-92f4-4dc0-8ac1-b03e509606c3" containerName="payload-processing"
Nov 02 01:51:47.536621 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.536589    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79481b1b-92f4-4dc0-8ac1-b03e509606c3" containerName="payload-processing"
Nov 02 01:51:47.537069 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.536741    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="79481b1b-92f4-4dc0-8ac1-b03e509606c3" containerName="payload-processing"
Nov 02 01:51:47.537069 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.536758    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef2ff06b-7835-4075-9346-7dc5ced29cf4" containerName="payload-pre-processing"
Nov 02 01:51:47.537069 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.536766    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccbdec1a-9065-4b40-bf67-21d4f8c30e33" containerName="payload-processing"
Nov 02 01:51:47.537069 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.536775    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a0ddab3-c6be-44e4-800c-012437a54a2e" containerName="payload-pre-processing"
Nov 02 01:51:47.537069 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.536790    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="405c8e1f-2f4b-4624-ac6c-5f607e4bb81d" containerName="payload-pre-processing"
Nov 02 01:51:47.537069 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.536797    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="514e63d2-f45e-4e85-aa53-1799f73f47ab" containerName="revoke-keys"
Nov 02 01:51:47.537069 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.536803    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="7392e99a-1aa8-48f9-ab30-4fea2eab88de" containerName="payload-processing"
Nov 02 01:51:47.542936 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.542907    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:47.595305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.594317    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-credential-socket\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:47.595305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.594430    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-token\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:47.595305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.594531    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-workload-socket\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:47.595305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.594651    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-envoy\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:47.595305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.594701    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-data\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:47.595305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.594730    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istiod-ca-cert\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:47.595305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.594804    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m52q\" (UniqueName: \"kubernetes.io/projected/d8ba9d0e-66ce-4927-aed2-e84b3280e116-kube-api-access-7m52q\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:47.595305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.594852    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-podinfo\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:47.595305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.594909    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-workload-certs\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:47.760320 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.683681    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-8ebd9167-gw-openshift-default-dockercfg-qmd46\""
Nov 02 01:51:48.049337 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:47.910822    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-token\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:48.074312 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:48.049715    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-workload-socket\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:48.074312 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:48.064444    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-envoy\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:48.074312 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:48.064520    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-data\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:48.074312 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:48.064558    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istiod-ca-cert\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:48.074312 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:48.064662    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m52q\" (UniqueName: \"kubernetes.io/projected/d8ba9d0e-66ce-4927-aed2-e84b3280e116-kube-api-access-7m52q\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:48.074312 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:48.064723    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-podinfo\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:48.074312 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:48.064804    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-workload-certs\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:48.074312 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:48.064954    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-credential-socket\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:48.074312 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:48.025444    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-token\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:48.096843 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:48.096806    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-workload-socket\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:48.175306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:48.170650    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-workload-certs\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:48.193720 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:48.186344    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-data\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:48.194456 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:48.194426    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istiod-ca-cert\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:48.198401 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:48.197351    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"]
Nov 02 01:51:48.200648 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:48.198649    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-credential-socket\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:48.219707 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:48.219642    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-envoy\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:48.234320 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:48.233555    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"]
Nov 02 01:51:48.234320 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:48.233837    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5" podUID="8b71baa1-9d31-48de-a3ec-379f6f5eae37" containerName="istio-proxy" containerID="cri-o://fc7d3e2d55b46e5000bb6ce2b590e7dd61e5f93b00b9458c0d8fca23b64fad30" gracePeriod=30
Nov 02 01:51:48.238558 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:48.238485    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m52q\" (UniqueName: \"kubernetes.io/projected/d8ba9d0e-66ce-4927-aed2-e84b3280e116-kube-api-access-7m52q\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:48.272403 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:48.263819    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-podinfo\") pod \"e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") " pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:48.461727 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:48.461602    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:48.592930 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:48.590819    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd723454-rgtxn"]
Nov 02 01:51:48.596408 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:48.596386    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd723454-rgtxn"]
Nov 02 01:51:49.040226 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:49.024251    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"]
Nov 02 01:51:49.048239 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:49.048185    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:51:49.048627 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:49.048501    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:51:49.048627 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:49.048576    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:51:49.240710 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:49.240671    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56b5b049-867f-4f13-90e5-d2b280044f2d" path="/var/lib/kubelet/pods/56b5b049-867f-4f13-90e5-d2b280044f2d/volumes"
Nov 02 01:51:49.241447 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:49.241424    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddc019a8-a65c-4f8a-9d98-a638232a10ec" path="/var/lib/kubelet/pods/ddc019a8-a65c-4f8a-9d98-a638232a10ec/volumes"
Nov 02 01:51:49.414133 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:49.414095    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb" event={"ID":"d8ba9d0e-66ce-4927-aed2-e84b3280e116","Type":"ContainerStarted","Data":"0a1c65adade57ed2ad17262856d7e3f5da351049750421a7b5efd8f0bdb10f26"}
Nov 02 01:51:49.414343 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:49.414149    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb" event={"ID":"d8ba9d0e-66ce-4927-aed2-e84b3280e116","Type":"ContainerStarted","Data":"df5bfeab3f9ab0da0399f54bf56a0e2cb68241e60fd154b3450e7d03e3b64a68"}
Nov 02 01:51:49.462922 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:49.462886    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:49.537500 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:49.537412    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb" podStartSLOduration=2.537398692 podStartE2EDuration="2.537398692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:51:49.53366504 +0000 UTC m=+1574.437347155" watchObservedRunningTime="2026-11-02 01:51:49.537398692 +0000 UTC m=+1574.441080871"
Nov 02 01:51:50.468956 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:50.468925    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:51.431718 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:51.431684    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:51.432617 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:51.432597    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:51:53.531181 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:53.529167    5639 generic.go:356] "Generic (PLEG): container finished" podID="8b71baa1-9d31-48de-a3ec-379f6f5eae37" containerID="fc7d3e2d55b46e5000bb6ce2b590e7dd61e5f93b00b9458c0d8fca23b64fad30" exitCode=0
Nov 02 01:51:53.531181 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:53.530583    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5" event={"ID":"8b71baa1-9d31-48de-a3ec-379f6f5eae37","Type":"ContainerDied","Data":"fc7d3e2d55b46e5000bb6ce2b590e7dd61e5f93b00b9458c0d8fca23b64fad30"}
Nov 02 01:51:53.531181 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:53.530649    5639 scope.go:122] "RemoveContainer" containerID="5107a59619749632b90e634b66f0b7ba6535efb394ae63ff6752b6f76fc7bbe0"
Nov 02 01:51:53.965720 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:53.965693    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:51:54.017166 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.017131    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istiod-ca-cert\") pod \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") "
Nov 02 01:51:54.017379 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.017249    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-data\") pod \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") "
Nov 02 01:51:54.017379 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.017356    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-workload-socket\") pod \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") "
Nov 02 01:51:54.017509 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.017499    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-token\" (UniqueName: \"kubernetes.io/projected/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-token\") pod \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") "
Nov 02 01:51:54.017588 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.017561    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istiod-ca-cert" pod "8b71baa1-9d31-48de-a3ec-379f6f5eae37" (UID: "8b71baa1-9d31-48de-a3ec-379f6f5eae37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:51:54.017661 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.017571    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-envoy\") pod \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") "
Nov 02 01:51:54.017721 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.017655    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-workload-socket" pod "8b71baa1-9d31-48de-a3ec-379f6f5eae37" (UID: "8b71baa1-9d31-48de-a3ec-379f6f5eae37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:51:54.017721 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.017710    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b71baa1-9d31-48de-a3ec-379f6f5eae37-kube-api-access-ln6w8\" (UniqueName: \"kubernetes.io/projected/8b71baa1-9d31-48de-a3ec-379f6f5eae37-kube-api-access-ln6w8\") pod \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") "
Nov 02 01:51:54.017881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.017802    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-credential-socket\") pod \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") "
Nov 02 01:51:54.017881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.017842    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-workload-certs\") pod \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") "
Nov 02 01:51:54.018005 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.017940    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-podinfo\") pod \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\" (UID: \"8b71baa1-9d31-48de-a3ec-379f6f5eae37\") "
Nov 02 01:51:54.018118 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.018092    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-credential-socket" pod "8b71baa1-9d31-48de-a3ec-379f6f5eae37" (UID: "8b71baa1-9d31-48de-a3ec-379f6f5eae37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:51:54.018462 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.018441    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-workload-certs" pod "8b71baa1-9d31-48de-a3ec-379f6f5eae37" (UID: "8b71baa1-9d31-48de-a3ec-379f6f5eae37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:51:54.018771 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.018748    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:54.018878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.018772    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:54.018878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.018787    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:54.018878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.018800    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:54.019056 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.017545    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-data" pod "8b71baa1-9d31-48de-a3ec-379f6f5eae37" (UID: "8b71baa1-9d31-48de-a3ec-379f6f5eae37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:51:54.020718 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.020684    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-envoy" pod "8b71baa1-9d31-48de-a3ec-379f6f5eae37" (UID: "8b71baa1-9d31-48de-a3ec-379f6f5eae37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:51:54.020812 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.020775    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-token" pod "8b71baa1-9d31-48de-a3ec-379f6f5eae37" (UID: "8b71baa1-9d31-48de-a3ec-379f6f5eae37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:51:54.020923 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.020906    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-podinfo" pod "8b71baa1-9d31-48de-a3ec-379f6f5eae37" (UID: "8b71baa1-9d31-48de-a3ec-379f6f5eae37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:51:54.022512 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.022485    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b71baa1-9d31-48de-a3ec-379f6f5eae37-kube-api-access-ln6w8" pod "8b71baa1-9d31-48de-a3ec-379f6f5eae37" (UID: "8b71baa1-9d31-48de-a3ec-379f6f5eae37"). InnerVolumeSpecName "kube-api-access-ln6w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:51:54.124859 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.124826    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:54.124859 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.124857    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:54.125071 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.124869    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln6w8\" (UniqueName: \"kubernetes.io/projected/8b71baa1-9d31-48de-a3ec-379f6f5eae37-kube-api-access-ln6w8\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:54.125071 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.124879    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:54.125071 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.124887    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b71baa1-9d31-48de-a3ec-379f6f5eae37-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:54.623027 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.621951    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5" event={"ID":"8b71baa1-9d31-48de-a3ec-379f6f5eae37","Type":"ContainerDied","Data":"6544d64e61a4d8e32cd392d4da5f1a4d627bb1d5ccf687044483fb1f9648a824"}
Nov 02 01:51:54.623027 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.622019    5639 scope.go:122] "RemoveContainer" containerID="fc7d3e2d55b46e5000bb6ce2b590e7dd61e5f93b00b9458c0d8fca23b64fad30"
Nov 02 01:51:54.623027 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.622172    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"
Nov 02 01:51:54.791749 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.791716    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"]
Nov 02 01:51:54.841048 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:54.841019    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2821b375-openshift-default-6b8bffbbd5-m72l5"]
Nov 02 01:51:55.221720 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:55.221688    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b71baa1-9d31-48de-a3ec-379f6f5eae37" path="/var/lib/kubelet/pods/8b71baa1-9d31-48de-a3ec-379f6f5eae37/volumes"
Nov 02 01:51:55.931179 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:55.931145    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fc56410-99dhc"]
Nov 02 01:51:55.976566 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:55.976532    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fc56410-99dhc"]
Nov 02 01:51:57.069623 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:57.069591    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f385a5f1-vkrgb"]
Nov 02 01:51:57.106222 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:57.106183    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f385a5f1-vkrgb"]
Nov 02 01:51:57.220870 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:57.220835    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="514e63d2-f45e-4e85-aa53-1799f73f47ab" path="/var/lib/kubelet/pods/514e63d2-f45e-4e85-aa53-1799f73f47ab/volumes"
Nov 02 01:51:57.221398 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:51:57.221382    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3ceca1f-fff5-4d36-9de2-aaf984edd17b" path="/var/lib/kubelet/pods/a3ceca1f-fff5-4d36-9de2-aaf984edd17b/volumes"
Nov 02 01:51:57.423990 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:51:57.423951    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:52:01.064553 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.064521    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"]
Nov 02 01:52:01.065268 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.065244    5639 cpu_manager.go:414] "containerMap: removing container" podUID="8b71baa1-9d31-48de-a3ec-379f6f5eae37" containerName="istio-proxy"
Nov 02 01:52:01.065268 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.065261    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b71baa1-9d31-48de-a3ec-379f6f5eae37" containerName="istio-proxy"
Nov 02 01:52:01.065268 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.065273    5639 cpu_manager.go:414] "containerMap: removing container" podUID="8b71baa1-9d31-48de-a3ec-379f6f5eae37" containerName="istio-proxy"
Nov 02 01:52:01.065268 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.065292    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b71baa1-9d31-48de-a3ec-379f6f5eae37" containerName="istio-proxy"
Nov 02 01:52:01.065545 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.065435    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b71baa1-9d31-48de-a3ec-379f6f5eae37" containerName="istio-proxy"
Nov 02 01:52:01.065545 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.065443    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b71baa1-9d31-48de-a3ec-379f6f5eae37" containerName="istio-proxy"
Nov 02 01:52:01.070925 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.070903    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.113444 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.113414    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"]
Nov 02 01:52:01.118196 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.118174    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.122266 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.122227    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.122471 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.122449    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr4vb\" (UniqueName: \"kubernetes.io/projected/565a8a37-83e6-45ef-9b22-d524cbd6231e-kube-api-access-pr4vb\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.122576 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.122559    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.122668 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.122651    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.122727 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.122712    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.122871 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.122845    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565a8a37-83e6-45ef-9b22-d524cbd6231e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.122922 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.122907    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.122973 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.122958    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.123027 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.123012    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.170994 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.170960    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"]
Nov 02 01:52:01.224656 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.224612    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565a8a37-83e6-45ef-9b22-d524cbd6231e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.224849 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.224687    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.224849 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.224738    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.224849 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.224774    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.224849 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.224820    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.225129 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.224857    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.225129 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.224906    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sxt2\" (UniqueName: \"kubernetes.io/projected/56a5cefb-fa4f-49bf-9213-3951fa917f74-kube-api-access-4sxt2\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.225129 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.224956    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.225129 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.225032    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.225129 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.225058    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.225412 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.225163    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.225412 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.225202    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.225412 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.225296    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr4vb\" (UniqueName: \"kubernetes.io/projected/565a8a37-83e6-45ef-9b22-d524cbd6231e-kube-api-access-pr4vb\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.225412 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.225341    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.225617 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.225417    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.225617 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.225468    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56a5cefb-fa4f-49bf-9213-3951fa917f74-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.225617 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.225518    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.225617 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.225582    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.229301 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.226087    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.229301 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.226830    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565a8a37-83e6-45ef-9b22-d524cbd6231e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.229301 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.227461    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.229301 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.227736    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.229301 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.228371    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.233229 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.233203    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.234058 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.234031    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.234953 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.234937    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"]
Nov 02 01:52:01.289071 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.289043    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.289219 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.289145    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr4vb\" (UniqueName: \"kubernetes.io/projected/565a8a37-83e6-45ef-9b22-d524cbd6231e-kube-api-access-pr4vb\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b22tt\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.327510 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.327480    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.327691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.327553    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.327691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.327595    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.327691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.327663    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.327884 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.327735    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56a5cefb-fa4f-49bf-9213-3951fa917f74-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.327972 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.327935    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.327972 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.327941    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.328097 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.327986    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.328097 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.328020    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.328205 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.328134    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.328205 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.328167    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sxt2\" (UniqueName: \"kubernetes.io/projected/56a5cefb-fa4f-49bf-9213-3951fa917f74-kube-api-access-4sxt2\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.328362 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.328208    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.328468 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.328445    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56a5cefb-fa4f-49bf-9213-3951fa917f74-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.328554 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.328536    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.330391 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.330367    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.330588 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.330571    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.385737 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.385707    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:01.389064 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.389040    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.389227 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.389206    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sxt2\" (UniqueName: \"kubernetes.io/projected/56a5cefb-fa4f-49bf-9213-3951fa917f74-kube-api-access-4sxt2\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kvb8\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.435868 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.435834    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:01.869683 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:52:01.869650    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a5cefb_fa4f_49bf_9213_3951fa917f74.slice/crio-101934cccb93ab141b9456abb611bf07013443c303f8a4fc2d8095e1ede4a05b WatchSource:0}: Error finding container 101934cccb93ab141b9456abb611bf07013443c303f8a4fc2d8095e1ede4a05b: Status 404 returned error can't find the container with id 101934cccb93ab141b9456abb611bf07013443c303f8a4fc2d8095e1ede4a05b
Nov 02 01:52:01.871080 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.871056    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"]
Nov 02 01:52:01.872754 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.872711    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:52:01.872850 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.872799    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:52:01.872850 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.872840    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:52:01.877666 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.877644    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"]
Nov 02 01:52:01.883844 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.883808    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:52:01.883933 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.883876    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:52:01.883933 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:01.883906    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:52:02.718628 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:02.718590    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt" event={"ID":"565a8a37-83e6-45ef-9b22-d524cbd6231e","Type":"ContainerStarted","Data":"240557fdef98eb9cefd0ef6c19e0c51291c576be8b58f3d94fe3eb4c15d60150"}
Nov 02 01:52:02.719119 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:02.718672    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt" event={"ID":"565a8a37-83e6-45ef-9b22-d524cbd6231e","Type":"ContainerStarted","Data":"8f861252c25cf7f4d7b62bda7da20167c8b885d72cd5ab535a8a4e226d194b8a"}
Nov 02 01:52:02.720372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:02.720344    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8" event={"ID":"56a5cefb-fa4f-49bf-9213-3951fa917f74","Type":"ContainerStarted","Data":"9382717286a1010c5cac676f38cd0c8008061f8fe4c1e3cfd43ce110dbfc8e00"}
Nov 02 01:52:02.720469 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:02.720386    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8" event={"ID":"56a5cefb-fa4f-49bf-9213-3951fa917f74","Type":"ContainerStarted","Data":"101934cccb93ab141b9456abb611bf07013443c303f8a4fc2d8095e1ede4a05b"}
Nov 02 01:52:03.015359 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:03.014645    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt" podStartSLOduration=2.012126368 podStartE2EDuration="2.012126368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:52:02.910895787 +0000 UTC m=+1587.814577901" watchObservedRunningTime="2026-11-02 01:52:03.012126368 +0000 UTC m=+1587.915808486"
Nov 02 01:52:03.396306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:03.393265    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:03.396306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:03.393475    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-b22tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 02 01:52:03.396306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:03.393522    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt" podUID="565a8a37-83e6-45ef-9b22-d524cbd6231e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 02 01:52:03.436841 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:03.436803    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:03.438032 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:03.438010    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 02 01:52:03.438089 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:03.438062    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8" podUID="56a5cefb-fa4f-49bf-9213-3951fa917f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 02 01:52:04.387744 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:04.387705    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-b22tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 02 01:52:04.388495 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:04.388463    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt" podUID="565a8a37-83e6-45ef-9b22-d524cbd6231e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 02 01:52:04.436701 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:04.436669    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 02 01:52:04.436931 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:04.436909    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8" podUID="56a5cefb-fa4f-49bf-9213-3951fa917f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 02 01:52:05.052233 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:05.052177    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8" podStartSLOduration=4.052161607 podStartE2EDuration="4.052161607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:52:03.083628525 +0000 UTC m=+1587.987310652" watchObservedRunningTime="2026-11-02 01:52:05.052161607 +0000 UTC m=+1589.955843724"
Nov 02 01:52:05.053138 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:05.053118    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c52101d3-vnp2p"]
Nov 02 01:52:05.057943 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:05.057919    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c52101d3-vnp2p"]
Nov 02 01:52:05.224056 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:05.224023    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66dd198e-ff0a-4b6b-9011-32adb132e77c" path="/var/lib/kubelet/pods/66dd198e-ff0a-4b6b-9011-32adb132e77c/volumes"
Nov 02 01:52:05.389306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:05.388723    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-b22tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 02 01:52:05.389306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:05.388806    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt" podUID="565a8a37-83e6-45ef-9b22-d524cbd6231e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 02 01:52:05.436986 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:05.436951    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 02 01:52:05.437168 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:05.437002    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8" podUID="56a5cefb-fa4f-49bf-9213-3951fa917f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 02 01:52:06.402774 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:06.402732    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-b22tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 02 01:52:06.410470 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:06.410423    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt" podUID="565a8a37-83e6-45ef-9b22-d524cbd6231e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 02 01:52:06.436700 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:06.436671    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 02 01:52:06.436864 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:06.436718    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8" podUID="56a5cefb-fa4f-49bf-9213-3951fa917f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 02 01:52:07.389752 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:07.389711    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-b22tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 02 01:52:07.390114 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:07.390078    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt" podUID="565a8a37-83e6-45ef-9b22-d524cbd6231e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 02 01:52:07.436742 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:07.436688    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 02 01:52:07.437165 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:07.436773    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8" podUID="56a5cefb-fa4f-49bf-9213-3951fa917f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 02 01:52:08.386691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:08.386649    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-b22tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 02 01:52:08.386875 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:08.386729    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt" podUID="565a8a37-83e6-45ef-9b22-d524cbd6231e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 02 01:52:08.436339 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:08.436213    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 02 01:52:08.436339 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:08.436265    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8" podUID="56a5cefb-fa4f-49bf-9213-3951fa917f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 02 01:52:09.386652 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:09.386613    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-b22tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 02 01:52:09.387185 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:09.386678    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt" podUID="565a8a37-83e6-45ef-9b22-d524cbd6231e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 02 01:52:09.437389 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:09.437272    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 02 01:52:09.437583 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:09.437435    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8" podUID="56a5cefb-fa4f-49bf-9213-3951fa917f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 02 01:52:10.488241 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:10.488209    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:10.489142 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:10.489115    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:10.489625 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:10.489610    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 01:52:10.510795 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:10.510751    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:10.511092 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:10.511066    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:10.511982 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:10.511939    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 01:52:17.676822 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:17.676787    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:52:20.058863 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:20.058810    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74e25620-mpqgc"]
Nov 02 01:52:20.111132 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:20.111089    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74e25620-mpqgc"]
Nov 02 01:52:21.336107 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:21.336066    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca4c83be-268c-4163-8709-61c64ad2e750" path="/var/lib/kubelet/pods/ca4c83be-268c-4163-8709-61c64ad2e750/volumes"
Nov 02 01:52:21.361121 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:21.344372    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:52:21.480445 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:21.480384    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:52:22.365713 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:22.363385    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:52:22.365713 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:22.363992    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:52:22.485346 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:22.464834    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:52:23.050504 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:23.050466    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-564a8351-h2snf"]
Nov 02 01:52:23.054945 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:23.054922    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-564a8351-h2snf"]
Nov 02 01:52:23.222180 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:23.222147    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c229fa69-e9fd-4d5c-bfca-979c4602a398" path="/var/lib/kubelet/pods/c229fa69-e9fd-4d5c-bfca-979c4602a398/volumes"
Nov 02 01:52:27.423224 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:52:27.423179    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:52:33.051924 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:33.051890    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bd4274e-nnf2h"]
Nov 02 01:52:33.101316 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:33.101262    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a237e48-44z8t"]
Nov 02 01:52:33.105693 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:33.105670    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bd4274e-nnf2h"]
Nov 02 01:52:33.109629 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:33.109602    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a237e48-44z8t"]
Nov 02 01:52:33.221532 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:33.221502    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0096d1bd-5a8d-44fd-be18-d3b269ea8107" path="/var/lib/kubelet/pods/0096d1bd-5a8d-44fd-be18-d3b269ea8107/volumes"
Nov 02 01:52:33.222050 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:33.222035    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a30fb1a7-0467-4adc-928f-6a4a2eadb714" path="/var/lib/kubelet/pods/a30fb1a7-0467-4adc-928f-6a4a2eadb714/volumes"
Nov 02 01:52:39.055076 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:39.055035    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ea01c7a-svz7r"]
Nov 02 01:52:39.098832 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:39.098799    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ea01c7a-svz7r"]
Nov 02 01:52:39.227455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:39.227415    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0" path="/var/lib/kubelet/pods/a8efc2f3-fb5d-4b69-b215-7557ee5bf6b0/volumes"
Nov 02 01:52:40.987167 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:40.987134    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d"]
Nov 02 01:52:41.003676 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:41.003630    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d"
Nov 02 01:52:41.089312 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:41.081103    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d"]
Nov 02 01:52:41.105242 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:41.096989    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-8ebd9167\""
Nov 02 01:52:41.106472 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:41.106417    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-8ebd9167-dockercfg-4fwfg\""
Nov 02 01:52:41.284399 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:41.264629    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1935b100-31ea-4710-8dad-aa9c4f3ef816-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d\" (UID: \"1935b100-31ea-4710-8dad-aa9c4f3ef816\") " pod="openshift-ingress/payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d"
Nov 02 01:52:41.284399 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:41.264690    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fz7r\" (UniqueName: \"kubernetes.io/projected/1935b100-31ea-4710-8dad-aa9c4f3ef816-kube-api-access-4fz7r\") pod \"payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d\" (UID: \"1935b100-31ea-4710-8dad-aa9c4f3ef816\") " pod="openshift-ingress/payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d"
Nov 02 01:52:41.386313 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:41.368673    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1935b100-31ea-4710-8dad-aa9c4f3ef816-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d\" (UID: \"1935b100-31ea-4710-8dad-aa9c4f3ef816\") " pod="openshift-ingress/payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d"
Nov 02 01:52:41.386653 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:41.386621    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fz7r\" (UniqueName: \"kubernetes.io/projected/1935b100-31ea-4710-8dad-aa9c4f3ef816-kube-api-access-4fz7r\") pod \"payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d\" (UID: \"1935b100-31ea-4710-8dad-aa9c4f3ef816\") " pod="openshift-ingress/payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d"
Nov 02 01:52:41.401427 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:41.401361    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1935b100-31ea-4710-8dad-aa9c4f3ef816-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d\" (UID: \"1935b100-31ea-4710-8dad-aa9c4f3ef816\") " pod="openshift-ingress/payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d"
Nov 02 01:52:41.520897 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:41.520869    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6"]
Nov 02 01:52:41.524359 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:41.524227    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fz7r\" (UniqueName: \"kubernetes.io/projected/1935b100-31ea-4710-8dad-aa9c4f3ef816-kube-api-access-4fz7r\") pod \"payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d\" (UID: \"1935b100-31ea-4710-8dad-aa9c4f3ef816\") " pod="openshift-ingress/payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d"
Nov 02 01:52:41.530121 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:41.530098    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6"
Nov 02 01:52:41.537879 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:41.537243    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6"]
Nov 02 01:52:41.624977 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:41.624939    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzxzs\" (UniqueName: \"kubernetes.io/projected/80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d-kube-api-access-xzxzs\") pod \"payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6\" (UID: \"80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6"
Nov 02 01:52:41.625398 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:41.625363    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6\" (UID: \"80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6"
Nov 02 01:52:41.705358 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:41.697624    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d"
Nov 02 01:52:41.748322 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:41.736081    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6\" (UID: \"80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6"
Nov 02 01:52:41.748322 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:41.736195    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzxzs\" (UniqueName: \"kubernetes.io/projected/80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d-kube-api-access-xzxzs\") pod \"payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6\" (UID: \"80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6"
Nov 02 01:52:41.748322 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:41.737584    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6\" (UID: \"80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6"
Nov 02 01:52:41.872305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:41.872051    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzxzs\" (UniqueName: \"kubernetes.io/projected/80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d-kube-api-access-xzxzs\") pod \"payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6\" (UID: \"80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6"
Nov 02 01:52:42.156690 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.156605    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6"
Nov 02 01:52:42.310901 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.310867    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"]
Nov 02 01:52:42.317779 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.317749    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.373036 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.373010    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-59842dd7-openshift-default-dockercfg-znlfl\""
Nov 02 01:52:42.418586 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.418511    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"]
Nov 02 01:52:42.493507 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.487029    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-workload-socket\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.493507 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.487167    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-data\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.493507 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.487310    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-envoy\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.493507 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.487388    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbk9p\" (UniqueName: \"kubernetes.io/projected/86b56f9b-4c16-412c-b170-76133ab7abb2-kube-api-access-pbk9p\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.493507 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.487457    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-token\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.493507 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.487489    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-podinfo\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.493507 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.487527    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b56f9b-4c16-412c-b170-76133ab7abb2-istiod-ca-cert\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.493507 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.487572    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-workload-certs\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.493507 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.487647    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-credential-socket\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.569520 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.568734    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d"]
Nov 02 01:52:42.600332 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.589264    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-podinfo\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.600332 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.589388    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b56f9b-4c16-412c-b170-76133ab7abb2-istiod-ca-cert\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.600332 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.589433    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-workload-certs\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.600332 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.589504    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-credential-socket\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.600332 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.589629    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-workload-socket\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.600332 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.589739    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-data\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.600332 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.589856    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-envoy\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.600332 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.589916    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbk9p\" (UniqueName: \"kubernetes.io/projected/86b56f9b-4c16-412c-b170-76133ab7abb2-kube-api-access-pbk9p\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.600332 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.589990    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-token\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.600332 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.592728    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-workload-socket\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.600332 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.593469    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b56f9b-4c16-412c-b170-76133ab7abb2-istiod-ca-cert\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.600332 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.593732    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-workload-certs\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.600332 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.593975    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-credential-socket\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.600332 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.594653    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-data\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.629314 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:52:42.627101    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1935b100_31ea_4710_8dad_aa9c4f3ef816.slice/crio-74c2985a6b5f20422247c100627399780be6848c6860a0f7a0f40a6167226d84 WatchSource:0}: Error finding container 74c2985a6b5f20422247c100627399780be6848c6860a0f7a0f40a6167226d84: Status 404 returned error can't find the container with id 74c2985a6b5f20422247c100627399780be6848c6860a0f7a0f40a6167226d84
Nov 02 01:52:42.644268 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.644203    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-podinfo\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.675260 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.675135    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-envoy\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.716901 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.705690    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbk9p\" (UniqueName: \"kubernetes.io/projected/86b56f9b-4c16-412c-b170-76133ab7abb2-kube-api-access-pbk9p\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.729272 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.727886    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-token\") pod \"e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") " pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:42.780483 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.780431    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6"]
Nov 02 01:52:42.832091 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.832061    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d" event={"ID":"1935b100-31ea-4710-8dad-aa9c4f3ef816","Type":"ContainerStarted","Data":"74c2985a6b5f20422247c100627399780be6848c6860a0f7a0f40a6167226d84"}
Nov 02 01:52:42.833799 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.833774    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6" event={"ID":"80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d","Type":"ContainerStarted","Data":"9fd8f5129d839fe76444ae0f217b5f4fee478d169e830b859c12204d8b2dc693"}
Nov 02 01:52:42.934853 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:42.934777    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:43.053807 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:43.053772    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-829106b1-lssps"]
Nov 02 01:52:43.058257 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:43.058233    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-829106b1-lssps"]
Nov 02 01:52:43.221089 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:43.221016    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f6d3a51-9aeb-4548-9f8d-7158c6b13541" path="/var/lib/kubelet/pods/0f6d3a51-9aeb-4548-9f8d-7158c6b13541/volumes"
Nov 02 01:52:43.372348 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:43.371401    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"]
Nov 02 01:52:43.372348 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:52:43.371524    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b56f9b_4c16_412c_b170_76133ab7abb2.slice/crio-5ceb8f7ba6d1ad9a9e3f4e14a824060d79a352c6f62f78b25951b48cb8173484 WatchSource:0}: Error finding container 5ceb8f7ba6d1ad9a9e3f4e14a824060d79a352c6f62f78b25951b48cb8173484: Status 404 returned error can't find the container with id 5ceb8f7ba6d1ad9a9e3f4e14a824060d79a352c6f62f78b25951b48cb8173484
Nov 02 01:52:43.376567 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:43.376529    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:52:43.376662 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:43.376601    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:52:43.376662 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:43.376630    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:52:43.532317 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:43.532175    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"]
Nov 02 01:52:43.532490 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:43.532445    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc" podUID="08c64df3-8d18-42b1-ba44-a8f0bc5ae652" containerName="istio-proxy" containerID="cri-o://496173330f9a8113138bcce1342a857bbb3086f1e4b5fd18d8aab537e335bf6e" gracePeriod=30
Nov 02 01:52:43.844692 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:43.844655    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6" event={"ID":"80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d","Type":"ContainerStarted","Data":"0bcbbe96df9e1f6763ed7d4834078e69e4cb419705e198dab89ca02880d6cc9b"}
Nov 02 01:52:43.845033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:43.844770    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6"
Nov 02 01:52:43.846883 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:43.846846    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d" event={"ID":"86b56f9b-4c16-412c-b170-76133ab7abb2","Type":"ContainerStarted","Data":"fbeb990c66f31886a87770784922ccdee559e57d9929090671b9a8fb659e3375"}
Nov 02 01:52:43.847014 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:43.846906    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d" event={"ID":"86b56f9b-4c16-412c-b170-76133ab7abb2","Type":"ContainerStarted","Data":"5ceb8f7ba6d1ad9a9e3f4e14a824060d79a352c6f62f78b25951b48cb8173484"}
Nov 02 01:52:43.849021 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:43.849000    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d" event={"ID":"1935b100-31ea-4710-8dad-aa9c4f3ef816","Type":"ContainerStarted","Data":"57cf9b0bf2ed7ecfab3ddedf5a243ecafa144c524bcedb6de8d2e992723e7a94"}
Nov 02 01:52:43.849144 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:43.849057    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d"
Nov 02 01:52:43.935434 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:43.935407    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:43.968839 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:43.968792    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6" podStartSLOduration=2.968777567 podStartE2EDuration="2.968777567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:52:43.965339002 +0000 UTC m=+1628.869021118" watchObservedRunningTime="2026-11-02 01:52:43.968777567 +0000 UTC m=+1628.872459681"
Nov 02 01:52:44.184875 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:44.178049    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d" podStartSLOduration=4.178025866 podStartE2EDuration="4.178025866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:52:44.101613967 +0000 UTC m=+1629.005296098" watchObservedRunningTime="2026-11-02 01:52:44.178025866 +0000 UTC m=+1629.081707981"
Nov 02 01:52:44.254042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:44.253609    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d" podStartSLOduration=2.253590432 podStartE2EDuration="2.253590432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:52:44.248436913 +0000 UTC m=+1629.152119051" watchObservedRunningTime="2026-11-02 01:52:44.253590432 +0000 UTC m=+1629.157272548"
Nov 02 01:52:44.940844 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:44.940816    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:45.866139 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:45.866112    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:45.867201 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:45.867181    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:52:47.882444 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:47.882406    5639 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:52:47.882843 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:47.882464    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc" podUID="08c64df3-8d18-42b1-ba44-a8f0bc5ae652" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:52:48.832920 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.832890    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:52:48.893970 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.893937    5639 generic.go:356] "Generic (PLEG): container finished" podID="08c64df3-8d18-42b1-ba44-a8f0bc5ae652" containerID="496173330f9a8113138bcce1342a857bbb3086f1e4b5fd18d8aab537e335bf6e" exitCode=0
Nov 02 01:52:48.894418 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.893987    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"
Nov 02 01:52:48.894418 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.893988    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc" event={"ID":"08c64df3-8d18-42b1-ba44-a8f0bc5ae652","Type":"ContainerDied","Data":"496173330f9a8113138bcce1342a857bbb3086f1e4b5fd18d8aab537e335bf6e"}
Nov 02 01:52:48.894418 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.894119    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc" event={"ID":"08c64df3-8d18-42b1-ba44-a8f0bc5ae652","Type":"ContainerDied","Data":"6093efd3e55a8d2f56c0304bc87ea03064d7276b46b6e6be5c04d525c5a03a6e"}
Nov 02 01:52:48.894418 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.894153    5639 scope.go:122] "RemoveContainer" containerID="496173330f9a8113138bcce1342a857bbb3086f1e4b5fd18d8aab537e335bf6e"
Nov 02 01:52:48.905680 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.905660    5639 scope.go:122] "RemoveContainer" containerID="496173330f9a8113138bcce1342a857bbb3086f1e4b5fd18d8aab537e335bf6e"
Nov 02 01:52:48.905966 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:52:48.905947    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"496173330f9a8113138bcce1342a857bbb3086f1e4b5fd18d8aab537e335bf6e\": container with ID starting with 496173330f9a8113138bcce1342a857bbb3086f1e4b5fd18d8aab537e335bf6e not found: ID does not exist" containerID="496173330f9a8113138bcce1342a857bbb3086f1e4b5fd18d8aab537e335bf6e"
Nov 02 01:52:48.906022 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.905975    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"496173330f9a8113138bcce1342a857bbb3086f1e4b5fd18d8aab537e335bf6e"} err="failed to get container status \"496173330f9a8113138bcce1342a857bbb3086f1e4b5fd18d8aab537e335bf6e\": rpc error: code = NotFound desc = could not find container \"496173330f9a8113138bcce1342a857bbb3086f1e4b5fd18d8aab537e335bf6e\": container with ID starting with 496173330f9a8113138bcce1342a857bbb3086f1e4b5fd18d8aab537e335bf6e not found: ID does not exist"
Nov 02 01:52:48.913201 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.913107    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istiod-ca-cert\") pod \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") "
Nov 02 01:52:48.913346 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.913202    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-token\" (UniqueName: \"kubernetes.io/projected/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-token\") pod \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") "
Nov 02 01:52:48.913423 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.913367    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-credential-socket\") pod \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") "
Nov 02 01:52:48.913499 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.913429    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-envoy\") pod \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") "
Nov 02 01:52:48.913551 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.913499    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-workload-socket\") pod \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") "
Nov 02 01:52:48.913551 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.913521    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istiod-ca-cert" pod "08c64df3-8d18-42b1-ba44-a8f0bc5ae652" (UID: "08c64df3-8d18-42b1-ba44-a8f0bc5ae652"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:52:48.913678 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.913646    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-kube-api-access-gcnrn\" (UniqueName: \"kubernetes.io/projected/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-kube-api-access-gcnrn\") pod \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") "
Nov 02 01:52:48.913744 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.913680    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-podinfo\") pod \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") "
Nov 02 01:52:48.913744 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.913737    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-data\") pod \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") "
Nov 02 01:52:48.913857 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.913767    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-workload-certs\") pod \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\" (UID: \"08c64df3-8d18-42b1-ba44-a8f0bc5ae652\") "
Nov 02 01:52:48.913914 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.913680    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-credential-socket" pod "08c64df3-8d18-42b1-ba44-a8f0bc5ae652" (UID: "08c64df3-8d18-42b1-ba44-a8f0bc5ae652"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:52:48.914942 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.914912    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:48.915038 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.914946    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:48.915183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.915163    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-workload-certs" pod "08c64df3-8d18-42b1-ba44-a8f0bc5ae652" (UID: "08c64df3-8d18-42b1-ba44-a8f0bc5ae652"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:52:48.915390 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.915368    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-data" pod "08c64df3-8d18-42b1-ba44-a8f0bc5ae652" (UID: "08c64df3-8d18-42b1-ba44-a8f0bc5ae652"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:52:48.915610 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.915585    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-workload-socket" pod "08c64df3-8d18-42b1-ba44-a8f0bc5ae652" (UID: "08c64df3-8d18-42b1-ba44-a8f0bc5ae652"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:52:48.922787 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.921705    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-token" pod "08c64df3-8d18-42b1-ba44-a8f0bc5ae652" (UID: "08c64df3-8d18-42b1-ba44-a8f0bc5ae652"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:52:48.923627 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.923472    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-envoy" pod "08c64df3-8d18-42b1-ba44-a8f0bc5ae652" (UID: "08c64df3-8d18-42b1-ba44-a8f0bc5ae652"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:52:48.923627 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.923564    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-podinfo" pod "08c64df3-8d18-42b1-ba44-a8f0bc5ae652" (UID: "08c64df3-8d18-42b1-ba44-a8f0bc5ae652"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:52:48.923627 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:48.923586    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-kube-api-access-gcnrn" pod "08c64df3-8d18-42b1-ba44-a8f0bc5ae652" (UID: "08c64df3-8d18-42b1-ba44-a8f0bc5ae652"). InnerVolumeSpecName "kube-api-access-gcnrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:52:49.016672 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:49.016636    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:49.016672 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:49.016671    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:49.016879 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:49.016687    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:49.016879 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:49.016701    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcnrn\" (UniqueName: \"kubernetes.io/projected/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-kube-api-access-gcnrn\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:49.016879 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:49.016715    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:49.016879 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:49.016729    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:49.016879 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:49.016742    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c64df3-8d18-42b1-ba44-a8f0bc5ae652-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:49.319523 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:49.319488    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"]
Nov 02 01:52:49.324547 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:49.324515    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-md4gc"]
Nov 02 01:52:51.221142 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:51.221106    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08c64df3-8d18-42b1-ba44-a8f0bc5ae652" path="/var/lib/kubelet/pods/08c64df3-8d18-42b1-ba44-a8f0bc5ae652/volumes"
Nov 02 01:52:54.860568 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:54.860532    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6"
Nov 02 01:52:54.860972 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:54.860604    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d"
Nov 02 01:52:57.051108 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:57.051072    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac9cf4a5-bzpjr"]
Nov 02 01:52:57.065391 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:57.065366    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac9cf4a5-bzpjr"]
Nov 02 01:52:57.100985 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:57.100956    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cdceff0-29qrc"]
Nov 02 01:52:57.109880 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:57.109841    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cdceff0-29qrc"]
Nov 02 01:52:57.220741 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:57.220706    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="174ead9f-a5b5-4025-bea8-3a18dbfab699" path="/var/lib/kubelet/pods/174ead9f-a5b5-4025-bea8-3a18dbfab699/volumes"
Nov 02 01:52:57.221258 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:57.221237    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b32bf8e-cc36-4898-97b0-dcda06e24f87" path="/var/lib/kubelet/pods/6b32bf8e-cc36-4898-97b0-dcda06e24f87/volumes"
Nov 02 01:52:57.422173 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:52:57.422137    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:52:58.015701 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:52:58.015671    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:53:00.049311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:00.049262    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5c9e24a-c7gkz"]
Nov 02 01:53:00.099499 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:00.099466    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5c9e24a-c7gkz"]
Nov 02 01:53:01.049546 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:01.049517    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-346c5448-s542p"]
Nov 02 01:53:01.099174 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:01.099140    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-346c5448-s542p"]
Nov 02 01:53:01.220653 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:01.220616    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05d86e93-5bbf-40ee-b1e8-a20c811378ba" path="/var/lib/kubelet/pods/05d86e93-5bbf-40ee-b1e8-a20c811378ba/volumes"
Nov 02 01:53:01.221138 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:01.221123    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0a5cb8e-fafb-42bc-a01a-fbb88873a88b" path="/var/lib/kubelet/pods/f0a5cb8e-fafb-42bc-a01a-fbb88873a88b/volumes"
Nov 02 01:53:05.055501 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:05.055464    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6ed0482-2mkqb"]
Nov 02 01:53:05.082006 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:05.081958    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6ed0482-2mkqb"]
Nov 02 01:53:05.264087 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:05.264050    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9726c116-9266-4a62-bfd7-3d58ddb6cf98" path="/var/lib/kubelet/pods/9726c116-9266-4a62-bfd7-3d58ddb6cf98/volumes"
Nov 02 01:53:07.473428 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:07.473390    5639 patch_prober.go:29] interesting pod/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:53:07.473935 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:07.473478    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb" podUID="d8ba9d0e-66ce-4927-aed2-e84b3280e116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:53:08.050429 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:08.050396    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f1b79b0-5q5hw"]
Nov 02 01:53:08.099672 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:08.099644    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f1b79b0-5q5hw"]
Nov 02 01:53:09.225530 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:09.225504    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f96ea7a-c7f4-429c-9c49-39a99091044f" path="/var/lib/kubelet/pods/2f96ea7a-c7f4-429c-9c49-39a99091044f/volumes"
Nov 02 01:53:10.111886 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:10.111854    5639 generic.go:356] "Generic (PLEG): container finished" podID="d8ba9d0e-66ce-4927-aed2-e84b3280e116" containerID="0a1c65adade57ed2ad17262856d7e3f5da351049750421a7b5efd8f0bdb10f26" exitCode=0
Nov 02 01:53:10.112097 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:10.111931    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb" event={"ID":"d8ba9d0e-66ce-4927-aed2-e84b3280e116","Type":"ContainerDied","Data":"0a1c65adade57ed2ad17262856d7e3f5da351049750421a7b5efd8f0bdb10f26"}
Nov 02 01:53:10.112490 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:10.112470    5639 scope.go:122] "RemoveContainer" containerID="0a1c65adade57ed2ad17262856d7e3f5da351049750421a7b5efd8f0bdb10f26"
Nov 02 01:53:10.114168 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:10.114141    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:53:10.114224 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:10.114211    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:53:10.114259 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:10.114244    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:53:11.123934 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:11.123903    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb" event={"ID":"d8ba9d0e-66ce-4927-aed2-e84b3280e116","Type":"ContainerStarted","Data":"d56a3ead53902f02de26e785569ad9adfd55f6517c54a1ddc8213f86c19f163e"}
Nov 02 01:53:11.464359 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:11.463664    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:53:11.464359 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:11.463808    5639 patch_prober.go:29] interesting pod/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 02 01:53:11.464359 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:11.463854    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb" podUID="d8ba9d0e-66ce-4927-aed2-e84b3280e116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 02 01:53:12.465972 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:12.465934    5639 patch_prober.go:29] interesting pod/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 02 01:53:12.470442 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:12.470405    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb" podUID="d8ba9d0e-66ce-4927-aed2-e84b3280e116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 02 01:53:13.466326 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:13.463652    5639 patch_prober.go:29] interesting pod/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 02 01:53:13.466326 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:13.463731    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb" podUID="d8ba9d0e-66ce-4927-aed2-e84b3280e116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 02 01:53:14.478312 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:14.471938    5639 patch_prober.go:29] interesting pod/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 02 01:53:14.478312 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:14.472016    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb" podUID="d8ba9d0e-66ce-4927-aed2-e84b3280e116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 02 01:53:15.484878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:15.484413    5639 patch_prober.go:29] interesting pod/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 02 01:53:15.484878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:15.484477    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb" podUID="d8ba9d0e-66ce-4927-aed2-e84b3280e116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 02 01:53:16.301416 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.301375    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"]
Nov 02 01:53:16.302600 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.302572    5639 cpu_manager.go:414] "containerMap: removing container" podUID="08c64df3-8d18-42b1-ba44-a8f0bc5ae652" containerName="istio-proxy"
Nov 02 01:53:16.302600 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.302599    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c64df3-8d18-42b1-ba44-a8f0bc5ae652" containerName="istio-proxy"
Nov 02 01:53:16.302890 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.302872    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c64df3-8d18-42b1-ba44-a8f0bc5ae652" containerName="istio-proxy"
Nov 02 01:53:16.306482 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.306456    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.380304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.369416    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-d700e9-openshift-default-dockercfg-skwgc\""
Nov 02 01:53:16.558049 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.557954    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2snh\" (UniqueName: \"kubernetes.io/projected/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-kube-api-access-z2snh\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.558668 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.558641    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-workload-socket\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.558814 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.558799    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-workload-certs\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.558923 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.558910    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-data\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.559092 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.559079    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-podinfo\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.563183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.563155    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-envoy\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.563585 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.563561    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-credential-socket\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.563763 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.563746    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-token\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.563919 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.563905    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istiod-ca-cert\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.681020 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.680328    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:53:16.681442 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.681418    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:53:16.689621 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.688312    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:53:16.701703 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.698765    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-credential-socket\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.701703 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.698831    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-token\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.701703 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.698891    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istiod-ca-cert\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.701703 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.699001    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2snh\" (UniqueName: \"kubernetes.io/projected/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-kube-api-access-z2snh\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.701703 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.699053    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-workload-socket\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.701703 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.699128    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-workload-certs\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.701703 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.699173    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-data\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.701703 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.699317    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-podinfo\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.701703 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.699345    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-envoy\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.701703 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.700672    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-credential-socket\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.703839 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.702750    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-workload-certs\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.703839 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.702896    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istiod-ca-cert\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.703839 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.703170    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-data\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.703839 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.703227    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"]
Nov 02 01:53:16.703839 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.703431    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-workload-socket\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.790663 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.790630    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-envoy\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.860021 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.859964    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-token\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.964304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.956604    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-podinfo\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.987566 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.965977    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2snh\" (UniqueName: \"kubernetes.io/projected/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-kube-api-access-z2snh\") pod \"e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") " pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:16.990815 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:16.990786    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:17.556851 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:53:17.556816    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceaab0ba_9b59_4a1a_839d_bbd4322f9629.slice/crio-9cb2374f3fe0bf66c64a0445ab1472f3ecdfe73471c3f6681fff1cec1923d423 WatchSource:0}: Error finding container 9cb2374f3fe0bf66c64a0445ab1472f3ecdfe73471c3f6681fff1cec1923d423: Status 404 returned error can't find the container with id 9cb2374f3fe0bf66c64a0445ab1472f3ecdfe73471c3f6681fff1cec1923d423
Nov 02 01:53:17.557388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:17.557256    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"]
Nov 02 01:53:17.559739 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:17.559695    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:53:17.560084 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:17.559784    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:53:17.560084 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:17.559818    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:53:18.303252 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:18.303213    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8" event={"ID":"ceaab0ba-9b59-4a1a-839d-bbd4322f9629","Type":"ContainerStarted","Data":"aff79c7033179a98e70ec2051073a3ee7ad68c5c722173160fa4124cb1f1135b"}
Nov 02 01:53:18.303558 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:18.303331    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8" event={"ID":"ceaab0ba-9b59-4a1a-839d-bbd4322f9629","Type":"ContainerStarted","Data":"9cb2374f3fe0bf66c64a0445ab1472f3ecdfe73471c3f6681fff1cec1923d423"}
Nov 02 01:53:18.429699 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:18.429640    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8" podStartSLOduration=2.429624969 podStartE2EDuration="2.429624969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:18.423375781 +0000 UTC m=+1663.327057915" watchObservedRunningTime="2026-11-02 01:53:18.429624969 +0000 UTC m=+1663.333307078"
Nov 02 01:53:18.992478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:18.992444    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:18.998237 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:18.998211    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:19.312433 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:19.312352    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:19.313414 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:19.313393    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:53:20.051119 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:20.051084    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c6366e-fs5gv"]
Nov 02 01:53:20.100558 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:20.100526    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c6366e-fs5gv"]
Nov 02 01:53:21.220858 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:21.220825    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8be73d62-1a59-4660-9515-aad1c170fb86" path="/var/lib/kubelet/pods/8be73d62-1a59-4660-9515-aad1c170fb86/volumes"
Nov 02 01:53:22.051865 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:22.051821    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76c31dd2-xt2x4"]
Nov 02 01:53:22.101995 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:22.101964    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76c31dd2-xt2x4"]
Nov 02 01:53:23.221263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:23.221232    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f02f9c71-728b-49af-9657-c3574c42f998" path="/var/lib/kubelet/pods/f02f9c71-728b-49af-9657-c3574c42f998/volumes"
Nov 02 01:53:25.050378 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:25.050349    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-372e00cd-kz2jm"]
Nov 02 01:53:25.099630 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:25.099598    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-372e00cd-kz2jm"]
Nov 02 01:53:25.222530 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:25.222497    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7abc26b-f2f9-49fa-afd1-d42f64c0b458" path="/var/lib/kubelet/pods/b7abc26b-f2f9-49fa-afd1-d42f64c0b458/volumes"
Nov 02 01:53:26.049874 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:26.049841    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c966814-z97g5"]
Nov 02 01:53:26.099143 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:26.099110    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c966814-z97g5"]
Nov 02 01:53:27.050543 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:27.050514    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f02408-f6kbp"]
Nov 02 01:53:27.100209 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:27.100175    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f02408-f6kbp"]
Nov 02 01:53:27.221191 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:27.221146    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="092d32f7-43a4-4b4d-a890-9941f86515e7" path="/var/lib/kubelet/pods/092d32f7-43a4-4b4d-a890-9941f86515e7/volumes"
Nov 02 01:53:27.221810 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:27.221786    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed7ff800-8da6-4dc6-9836-84affd85382a" path="/var/lib/kubelet/pods/ed7ff800-8da6-4dc6-9836-84affd85382a/volumes"
Nov 02 01:53:27.421160 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:53:27.421122    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:53:29.031917 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:29.031886    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd"]
Nov 02 01:53:29.036086 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:29.036069    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd"
Nov 02 01:53:29.089178 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:29.089130    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:53:29.138064 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:29.138036    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:53:29.138064 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:29.138032    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:53:29.138262 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:29.138218    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:53:29.143719 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:29.143695    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd"]
Nov 02 01:53:29.173829 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:29.173796    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd\" (UID: \"4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd"
Nov 02 01:53:29.174019 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:29.173908    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qtpg\" (UniqueName: \"kubernetes.io/projected/4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758-kube-api-access-8qtpg\") pod \"maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd\" (UID: \"4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd"
Nov 02 01:53:29.174019 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:29.173994    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd\" (UID: \"4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd"
Nov 02 01:53:29.276227 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:29.276185    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd\" (UID: \"4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd"
Nov 02 01:53:29.276512 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:29.276497    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd\" (UID: \"4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd"
Nov 02 01:53:29.276573 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:29.276538    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qtpg\" (UniqueName: \"kubernetes.io/projected/4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758-kube-api-access-8qtpg\") pod \"maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd\" (UID: \"4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd"
Nov 02 01:53:29.277104 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:29.277079    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd\" (UID: \"4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd"
Nov 02 01:53:29.278815 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:29.278784    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd\" (UID: \"4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd"
Nov 02 01:53:29.337653 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:29.337625    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qtpg\" (UniqueName: \"kubernetes.io/projected/4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758-kube-api-access-8qtpg\") pod \"maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd\" (UID: \"4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd"
Nov 02 01:53:29.351402 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:29.351378    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd"
Nov 02 01:53:29.841699 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:29.841663    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd"]
Nov 02 01:53:30.414208 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:30.414173    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd" event={"ID":"4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758","Type":"ContainerStarted","Data":"c5d78b841a6509a7bc6aedb22d11518db87b5bf98310d53e361879d4a818cf14"}
Nov 02 01:53:30.414637 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:30.414226    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd" event={"ID":"4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758","Type":"ContainerStarted","Data":"bf992a6c652309f0e382b59ffe38519cb3f2fb2eed5cd667d52d05e9c9ed1c7d"}
Nov 02 01:53:30.531795 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:30.531690    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd" podStartSLOduration=2.531675655 podStartE2EDuration="2.531675655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:30.529821534 +0000 UTC m=+1675.433503707" watchObservedRunningTime="2026-11-02 01:53:30.531675655 +0000 UTC m=+1675.435357767"
Nov 02 01:53:31.424219 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:31.424182    5639 generic.go:356] "Generic (PLEG): container finished" podID="4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758" containerID="c5d78b841a6509a7bc6aedb22d11518db87b5bf98310d53e361879d4a818cf14" exitCode=0
Nov 02 01:53:31.424645 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:31.424269    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd" event={"ID":"4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758","Type":"ContainerDied","Data":"c5d78b841a6509a7bc6aedb22d11518db87b5bf98310d53e361879d4a818cf14"}
Nov 02 01:53:32.701757 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:32.701515    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd"
Nov 02 01:53:32.784014 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:32.780194    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758-kube-api-access-8qtpg\" (UniqueName: \"kubernetes.io/projected/4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758-kube-api-access-8qtpg\") pod \"4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758\" (UID: \"4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758\") "
Nov 02 01:53:32.784014 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:32.780430    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758-maas-api-openshift-ca\") pod \"4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758\" (UID: \"4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758\") "
Nov 02 01:53:32.784014 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:32.780654    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758-maas-api-xks-ca\") pod \"4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758\" (UID: \"4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758\") "
Nov 02 01:53:32.829347 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:32.827902    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758-maas-api-openshift-ca" pod "4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758" (UID: "4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:53:32.829347 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:32.828122    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758-kube-api-access-8qtpg" pod "4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758" (UID: "4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758"). InnerVolumeSpecName "kube-api-access-8qtpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:53:32.910567 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:32.901949    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758-maas-api-xks-ca" pod "4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758" (UID: "4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:53:32.910567 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:32.906740    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:32.910567 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:32.906771    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:32.910567 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:32.906788    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qtpg\" (UniqueName: \"kubernetes.io/projected/4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758-kube-api-access-8qtpg\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:33.441765 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:33.441737    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd"
Nov 02 01:53:33.441765 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:33.441744    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd" event={"ID":"4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758","Type":"ContainerDied","Data":"bf992a6c652309f0e382b59ffe38519cb3f2fb2eed5cd667d52d05e9c9ed1c7d"}
Nov 02 01:53:33.441996 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:33.441801    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf992a6c652309f0e382b59ffe38519cb3f2fb2eed5cd667d52d05e9c9ed1c7d"
Nov 02 01:53:36.112539 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.112510    5639 scope.go:122] "RemoveContainer" containerID="d513affc5234441b3b8b09e0498c132904b32e417e5621113b949cb682f310ea"
Nov 02 01:53:36.154928 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.154903    5639 scope.go:122] "RemoveContainer" containerID="1d0feb417424c7c7c318cf627dfad88b531bb3f1e0b8f645ec90a8654caedcf0"
Nov 02 01:53:36.265314 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.253421    5639 scope.go:122] "RemoveContainer" containerID="10f596c2764d773d7680158b311e4b34000375428da7de77d4af2380065ad524"
Nov 02 01:53:36.368905 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.368807    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh"]
Nov 02 01:53:36.556326 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.542944    5639 cpu_manager.go:414] "containerMap: removing container" podUID="4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758" containerName="revoke-keys"
Nov 02 01:53:36.556326 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.542975    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758" containerName="revoke-keys"
Nov 02 01:53:36.556326 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.543354    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758" containerName="revoke-keys"
Nov 02 01:53:36.698630 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.693694    5639 scope.go:122] "RemoveContainer" containerID="52aa9def12c3debacf4c280676b12efd523e2e4a212900ccc0cd269b217aa1f9"
Nov 02 01:53:36.698630 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.694478    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh"
Nov 02 01:53:36.698630 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.696213    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh"]
Nov 02 01:53:36.714171 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.707153    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n"]
Nov 02 01:53:36.733640 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.730028    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n"]
Nov 02 01:53:36.733640 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.730154    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n"
Nov 02 01:53:36.758310 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.752748    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-59842dd7-dockercfg-859px\""
Nov 02 01:53:36.758310 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.752833    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-59842dd7\""
Nov 02 01:53:36.825536 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.825498    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxwsj\" (UniqueName: \"kubernetes.io/projected/b65549a1-6aa9-46a7-a2e7-1407ff736f44-kube-api-access-kxwsj\") pod \"payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n\" (UID: \"b65549a1-6aa9-46a7-a2e7-1407ff736f44\") " pod="openshift-ingress/payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n"
Nov 02 01:53:36.825823 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.825804    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eba17c91-872f-468d-b14d-422f3fb73b4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh\" (UID: \"eba17c91-872f-468d-b14d-422f3fb73b4f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh"
Nov 02 01:53:36.826034 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.826017    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65549a1-6aa9-46a7-a2e7-1407ff736f44-plugins-config-volume\") pod \"payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n\" (UID: \"b65549a1-6aa9-46a7-a2e7-1407ff736f44\") " pod="openshift-ingress/payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n"
Nov 02 01:53:36.826217 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.826202    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m59h\" (UniqueName: \"kubernetes.io/projected/eba17c91-872f-468d-b14d-422f3fb73b4f-kube-api-access-5m59h\") pod \"payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh\" (UID: \"eba17c91-872f-468d-b14d-422f3fb73b4f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh"
Nov 02 01:53:36.888721 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.885760    5639 scope.go:122] "RemoveContainer" containerID="5ce026485571e5be2f3db4ef87bed6fecd2af03cf296d75cf91c439e4c742916"
Nov 02 01:53:36.904590 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.904544    5639 scope.go:122] "RemoveContainer" containerID="326304970a3dd70a73795dd5ab7db92ae44eba237386820c7d2983c7f0ec001c"
Nov 02 01:53:36.919971 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.919946    5639 scope.go:122] "RemoveContainer" containerID="634d515bd66acba64c7daca22ac3d869a45236cf37435732c68ab70b4a3ca255"
Nov 02 01:53:36.933444 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.927790    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxwsj\" (UniqueName: \"kubernetes.io/projected/b65549a1-6aa9-46a7-a2e7-1407ff736f44-kube-api-access-kxwsj\") pod \"payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n\" (UID: \"b65549a1-6aa9-46a7-a2e7-1407ff736f44\") " pod="openshift-ingress/payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n"
Nov 02 01:53:36.933444 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.927865    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eba17c91-872f-468d-b14d-422f3fb73b4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh\" (UID: \"eba17c91-872f-468d-b14d-422f3fb73b4f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh"
Nov 02 01:53:36.933444 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.927970    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65549a1-6aa9-46a7-a2e7-1407ff736f44-plugins-config-volume\") pod \"payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n\" (UID: \"b65549a1-6aa9-46a7-a2e7-1407ff736f44\") " pod="openshift-ingress/payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n"
Nov 02 01:53:36.933444 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.928064    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m59h\" (UniqueName: \"kubernetes.io/projected/eba17c91-872f-468d-b14d-422f3fb73b4f-kube-api-access-5m59h\") pod \"payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh\" (UID: \"eba17c91-872f-468d-b14d-422f3fb73b4f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh"
Nov 02 01:53:36.933444 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.929839    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eba17c91-872f-468d-b14d-422f3fb73b4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh\" (UID: \"eba17c91-872f-468d-b14d-422f3fb73b4f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh"
Nov 02 01:53:36.933444 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.930707    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65549a1-6aa9-46a7-a2e7-1407ff736f44-plugins-config-volume\") pod \"payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n\" (UID: \"b65549a1-6aa9-46a7-a2e7-1407ff736f44\") " pod="openshift-ingress/payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n"
Nov 02 01:53:36.989688 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.989666    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxwsj\" (UniqueName: \"kubernetes.io/projected/b65549a1-6aa9-46a7-a2e7-1407ff736f44-kube-api-access-kxwsj\") pod \"payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n\" (UID: \"b65549a1-6aa9-46a7-a2e7-1407ff736f44\") " pod="openshift-ingress/payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n"
Nov 02 01:53:36.990043 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:36.990021    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m59h\" (UniqueName: \"kubernetes.io/projected/eba17c91-872f-468d-b14d-422f3fb73b4f-kube-api-access-5m59h\") pod \"payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh\" (UID: \"eba17c91-872f-468d-b14d-422f3fb73b4f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh"
Nov 02 01:53:37.170167 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:37.170140    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n"
Nov 02 01:53:37.174556 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:37.174534    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh"
Nov 02 01:53:37.561708 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:53:37.561678    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65549a1_6aa9_46a7_a2e7_1407ff736f44.slice/crio-5aa557112404d04506868591127e062d3c157cf30e87eb89b254d74fbcaa11c9 WatchSource:0}: Error finding container 5aa557112404d04506868591127e062d3c157cf30e87eb89b254d74fbcaa11c9: Status 404 returned error can't find the container with id 5aa557112404d04506868591127e062d3c157cf30e87eb89b254d74fbcaa11c9
Nov 02 01:53:37.562447 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:37.562344    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n"]
Nov 02 01:53:37.612232 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:37.612205    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh"]
Nov 02 01:53:37.612506 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:53:37.612484    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba17c91_872f_468d_b14d_422f3fb73b4f.slice/crio-a2756bbb63a5d448b9194c44dc88c476e9e792471c1ddd09a1065d7494d87c50 WatchSource:0}: Error finding container a2756bbb63a5d448b9194c44dc88c476e9e792471c1ddd09a1065d7494d87c50: Status 404 returned error can't find the container with id a2756bbb63a5d448b9194c44dc88c476e9e792471c1ddd09a1065d7494d87c50
Nov 02 01:53:37.744875 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:37.744842    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh" event={"ID":"eba17c91-872f-468d-b14d-422f3fb73b4f","Type":"ContainerStarted","Data":"79fd48e08e9c9067d982849afa6de82b434d56ed08c58e349ef4b8ce95d5b740"}
Nov 02 01:53:37.745045 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:37.744900    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh" event={"ID":"eba17c91-872f-468d-b14d-422f3fb73b4f","Type":"ContainerStarted","Data":"a2756bbb63a5d448b9194c44dc88c476e9e792471c1ddd09a1065d7494d87c50"}
Nov 02 01:53:37.745045 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:37.744932    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh"
Nov 02 01:53:37.746823 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:37.746794    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n" event={"ID":"b65549a1-6aa9-46a7-a2e7-1407ff736f44","Type":"ContainerStarted","Data":"e13e4fec26d258e921718bf15e0d0242162f30e9e287c3799ca0607aab0c164c"}
Nov 02 01:53:37.746939 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:37.746834    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n" event={"ID":"b65549a1-6aa9-46a7-a2e7-1407ff736f44","Type":"ContainerStarted","Data":"5aa557112404d04506868591127e062d3c157cf30e87eb89b254d74fbcaa11c9"}
Nov 02 01:53:37.746939 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:37.746864    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n"
Nov 02 01:53:37.864989 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:37.864934    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh" podStartSLOduration=1.8649164969999998 podStartE2EDuration="1.864916497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:37.860939258 +0000 UTC m=+1682.764621371" watchObservedRunningTime="2026-11-02 01:53:37.864916497 +0000 UTC m=+1682.768598638"
Nov 02 01:53:38.017927 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:38.017775    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n" podStartSLOduration=2.01775163 podStartE2EDuration="2.01775163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:37.992325071 +0000 UTC m=+1682.896007203" watchObservedRunningTime="2026-11-02 01:53:38.01775163 +0000 UTC m=+1682.921433744"
Nov 02 01:53:39.050368 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:39.050334    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ecd9882-r6t4x"]
Nov 02 01:53:39.101388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:39.101359    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ecd9882-r6t4x"]
Nov 02 01:53:39.190162 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:39.190125    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b1d120c-67n6k"]
Nov 02 01:53:39.193581 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:39.193561    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1d120c-67n6k"
Nov 02 01:53:39.221152 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:39.221121    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc2e0035-2bb2-47cf-a07a-78c060dc9ee3" path="/var/lib/kubelet/pods/dc2e0035-2bb2-47cf-a07a-78c060dc9ee3/volumes"
Nov 02 01:53:39.246790 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:39.246757    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:53:39.247336 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:39.247313    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:53:39.252668 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:39.252635    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b1d120c-67n6k"]
Nov 02 01:53:39.285408 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:39.285369    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76863b48-cafa-4e55-adb6-f98efc604043-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b1d120c-67n6k\" (UID: \"76863b48-cafa-4e55-adb6-f98efc604043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1d120c-67n6k"
Nov 02 01:53:39.285593 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:39.285464    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbkm2\" (UniqueName: \"kubernetes.io/projected/76863b48-cafa-4e55-adb6-f98efc604043-kube-api-access-xbkm2\") pod \"maas-api-revoke-sub-4b1d120c-67n6k\" (UID: \"76863b48-cafa-4e55-adb6-f98efc604043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1d120c-67n6k"
Nov 02 01:53:39.285593 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:39.285531    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76863b48-cafa-4e55-adb6-f98efc604043-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b1d120c-67n6k\" (UID: \"76863b48-cafa-4e55-adb6-f98efc604043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1d120c-67n6k"
Nov 02 01:53:39.296138 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:39.296112    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:53:39.296138 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:39.296128    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:53:39.387571 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:39.387533    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76863b48-cafa-4e55-adb6-f98efc604043-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b1d120c-67n6k\" (UID: \"76863b48-cafa-4e55-adb6-f98efc604043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1d120c-67n6k"
Nov 02 01:53:39.387761 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:39.387637    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbkm2\" (UniqueName: \"kubernetes.io/projected/76863b48-cafa-4e55-adb6-f98efc604043-kube-api-access-xbkm2\") pod \"maas-api-revoke-sub-4b1d120c-67n6k\" (UID: \"76863b48-cafa-4e55-adb6-f98efc604043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1d120c-67n6k"
Nov 02 01:53:39.387761 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:39.387668    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76863b48-cafa-4e55-adb6-f98efc604043-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b1d120c-67n6k\" (UID: \"76863b48-cafa-4e55-adb6-f98efc604043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1d120c-67n6k"
Nov 02 01:53:39.388417 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:39.388393    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76863b48-cafa-4e55-adb6-f98efc604043-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b1d120c-67n6k\" (UID: \"76863b48-cafa-4e55-adb6-f98efc604043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1d120c-67n6k"
Nov 02 01:53:39.390383 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:39.390358    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76863b48-cafa-4e55-adb6-f98efc604043-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b1d120c-67n6k\" (UID: \"76863b48-cafa-4e55-adb6-f98efc604043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1d120c-67n6k"
Nov 02 01:53:39.448788 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:39.448759    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbkm2\" (UniqueName: \"kubernetes.io/projected/76863b48-cafa-4e55-adb6-f98efc604043-kube-api-access-xbkm2\") pod \"maas-api-revoke-sub-4b1d120c-67n6k\" (UID: \"76863b48-cafa-4e55-adb6-f98efc604043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1d120c-67n6k"
Nov 02 01:53:39.507379 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:39.507354    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1d120c-67n6k"
Nov 02 01:53:39.892213 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:39.892188    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b1d120c-67n6k"]
Nov 02 01:53:39.892820 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:53:39.892794    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76863b48_cafa_4e55_adb6_f98efc604043.slice/crio-d80817daf9d50ac7634958b6efd64c69dd5b74a8c0e6958833976b03d26653d5 WatchSource:0}: Error finding container d80817daf9d50ac7634958b6efd64c69dd5b74a8c0e6958833976b03d26653d5: Status 404 returned error can't find the container with id d80817daf9d50ac7634958b6efd64c69dd5b74a8c0e6958833976b03d26653d5
Nov 02 01:53:40.050002 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:40.049972    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b228babd-49ckm"]
Nov 02 01:53:40.053934 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:40.053912    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b228babd-49ckm"]
Nov 02 01:53:40.505365 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:40.505334    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d"]
Nov 02 01:53:40.631187 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:40.631156    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6"]
Nov 02 01:53:40.774598 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:40.774552    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1d120c-67n6k" event={"ID":"76863b48-cafa-4e55-adb6-f98efc604043","Type":"ContainerStarted","Data":"9c51f0544405a16a33b268072e38d2f87d75ce7b90c3962ad208241edeca117c"}
Nov 02 01:53:40.774772 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:40.774651    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1d120c-67n6k" event={"ID":"76863b48-cafa-4e55-adb6-f98efc604043","Type":"ContainerStarted","Data":"d80817daf9d50ac7634958b6efd64c69dd5b74a8c0e6958833976b03d26653d5"}
Nov 02 01:53:40.893119 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:40.893032    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1d120c-67n6k" podStartSLOduration=1.893017951 podStartE2EDuration="1.893017951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:40.890074529 +0000 UTC m=+1685.793756678" watchObservedRunningTime="2026-11-02 01:53:40.893017951 +0000 UTC m=+1685.796700063"
Nov 02 01:53:41.275744 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:41.275670    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30d6f81a-329d-4cb7-b131-27a605f0ad07" path="/var/lib/kubelet/pods/30d6f81a-329d-4cb7-b131-27a605f0ad07/volumes"
Nov 02 01:53:41.784557 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:41.784520    5639 generic.go:356] "Generic (PLEG): container finished" podID="76863b48-cafa-4e55-adb6-f98efc604043" containerID="9c51f0544405a16a33b268072e38d2f87d75ce7b90c3962ad208241edeca117c" exitCode=0
Nov 02 01:53:41.784760 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:41.784569    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1d120c-67n6k" event={"ID":"76863b48-cafa-4e55-adb6-f98efc604043","Type":"ContainerDied","Data":"9c51f0544405a16a33b268072e38d2f87d75ce7b90c3962ad208241edeca117c"}
Nov 02 01:53:42.983348 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:42.983322    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1d120c-67n6k"
Nov 02 01:53:43.095973 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:43.095941    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76863b48-cafa-4e55-adb6-f98efc604043-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76863b48-cafa-4e55-adb6-f98efc604043-maas-api-openshift-ca\") pod \"76863b48-cafa-4e55-adb6-f98efc604043\" (UID: \"76863b48-cafa-4e55-adb6-f98efc604043\") "
Nov 02 01:53:43.096148 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:43.096020    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76863b48-cafa-4e55-adb6-f98efc604043-kube-api-access-xbkm2\" (UniqueName: \"kubernetes.io/projected/76863b48-cafa-4e55-adb6-f98efc604043-kube-api-access-xbkm2\") pod \"76863b48-cafa-4e55-adb6-f98efc604043\" (UID: \"76863b48-cafa-4e55-adb6-f98efc604043\") "
Nov 02 01:53:43.096370 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:43.096348    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76863b48-cafa-4e55-adb6-f98efc604043-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76863b48-cafa-4e55-adb6-f98efc604043-maas-api-xks-ca\") pod \"76863b48-cafa-4e55-adb6-f98efc604043\" (UID: \"76863b48-cafa-4e55-adb6-f98efc604043\") "
Nov 02 01:53:43.096456 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:43.096402    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76863b48-cafa-4e55-adb6-f98efc604043-maas-api-openshift-ca" pod "76863b48-cafa-4e55-adb6-f98efc604043" (UID: "76863b48-cafa-4e55-adb6-f98efc604043"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:53:43.097156 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:43.097138    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76863b48-cafa-4e55-adb6-f98efc604043-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:43.098631 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:43.098533    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76863b48-cafa-4e55-adb6-f98efc604043-kube-api-access-xbkm2" pod "76863b48-cafa-4e55-adb6-f98efc604043" (UID: "76863b48-cafa-4e55-adb6-f98efc604043"). InnerVolumeSpecName "kube-api-access-xbkm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:53:43.099073 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:43.099022    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76863b48-cafa-4e55-adb6-f98efc604043-maas-api-xks-ca" pod "76863b48-cafa-4e55-adb6-f98efc604043" (UID: "76863b48-cafa-4e55-adb6-f98efc604043"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:53:43.199506 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:43.199472    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76863b48-cafa-4e55-adb6-f98efc604043-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:43.199506 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:43.199500    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbkm2\" (UniqueName: \"kubernetes.io/projected/76863b48-cafa-4e55-adb6-f98efc604043-kube-api-access-xbkm2\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:43.802552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:43.802522    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1d120c-67n6k"
Nov 02 01:53:43.802552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:43.802528    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1d120c-67n6k" event={"ID":"76863b48-cafa-4e55-adb6-f98efc604043","Type":"ContainerDied","Data":"d80817daf9d50ac7634958b6efd64c69dd5b74a8c0e6958833976b03d26653d5"}
Nov 02 01:53:43.802835 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:43.802584    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d80817daf9d50ac7634958b6efd64c69dd5b74a8c0e6958833976b03d26653d5"
Nov 02 01:53:45.506121 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:45.506080    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d" podUID="1935b100-31ea-4710-8dad-aa9c4f3ef816" containerName="payload-processing" containerID="cri-o://57cf9b0bf2ed7ecfab3ddedf5a243ecafa144c524bcedb6de8d2e992723e7a94" gracePeriod=25
Nov 02 01:53:45.631719 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:45.631684    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6" podUID="80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d" containerName="payload-pre-processing" containerID="cri-o://0bcbbe96df9e1f6763ed7d4834078e69e4cb419705e198dab89ca02880d6cc9b" gracePeriod=25
Nov 02 01:53:45.801583 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:45.801554    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d"
Nov 02 01:53:45.829799 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:45.829763    5639 generic.go:356] "Generic (PLEG): container finished" podID="80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d" containerID="0bcbbe96df9e1f6763ed7d4834078e69e4cb419705e198dab89ca02880d6cc9b" exitCode=0
Nov 02 01:53:45.829962 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:45.829837    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6" event={"ID":"80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d","Type":"ContainerDied","Data":"0bcbbe96df9e1f6763ed7d4834078e69e4cb419705e198dab89ca02880d6cc9b"}
Nov 02 01:53:45.831905 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:45.831880    5639 generic.go:356] "Generic (PLEG): container finished" podID="1935b100-31ea-4710-8dad-aa9c4f3ef816" containerID="57cf9b0bf2ed7ecfab3ddedf5a243ecafa144c524bcedb6de8d2e992723e7a94" exitCode=0
Nov 02 01:53:45.832052 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:45.831937    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d"
Nov 02 01:53:45.832052 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:45.831963    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d" event={"ID":"1935b100-31ea-4710-8dad-aa9c4f3ef816","Type":"ContainerDied","Data":"57cf9b0bf2ed7ecfab3ddedf5a243ecafa144c524bcedb6de8d2e992723e7a94"}
Nov 02 01:53:45.832052 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:45.832030    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d" event={"ID":"1935b100-31ea-4710-8dad-aa9c4f3ef816","Type":"ContainerDied","Data":"74c2985a6b5f20422247c100627399780be6848c6860a0f7a0f40a6167226d84"}
Nov 02 01:53:45.832216 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:45.832059    5639 scope.go:122] "RemoveContainer" containerID="57cf9b0bf2ed7ecfab3ddedf5a243ecafa144c524bcedb6de8d2e992723e7a94"
Nov 02 01:53:45.843598 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:45.843580    5639 scope.go:122] "RemoveContainer" containerID="57cf9b0bf2ed7ecfab3ddedf5a243ecafa144c524bcedb6de8d2e992723e7a94"
Nov 02 01:53:45.843962 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:53:45.843911    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57cf9b0bf2ed7ecfab3ddedf5a243ecafa144c524bcedb6de8d2e992723e7a94\": container with ID starting with 57cf9b0bf2ed7ecfab3ddedf5a243ecafa144c524bcedb6de8d2e992723e7a94 not found: ID does not exist" containerID="57cf9b0bf2ed7ecfab3ddedf5a243ecafa144c524bcedb6de8d2e992723e7a94"
Nov 02 01:53:45.844088 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:45.843972    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57cf9b0bf2ed7ecfab3ddedf5a243ecafa144c524bcedb6de8d2e992723e7a94"} err="failed to get container status \"57cf9b0bf2ed7ecfab3ddedf5a243ecafa144c524bcedb6de8d2e992723e7a94\": rpc error: code = NotFound desc = could not find container \"57cf9b0bf2ed7ecfab3ddedf5a243ecafa144c524bcedb6de8d2e992723e7a94\": container with ID starting with 57cf9b0bf2ed7ecfab3ddedf5a243ecafa144c524bcedb6de8d2e992723e7a94 not found: ID does not exist"
Nov 02 01:53:45.942693 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:45.942661    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6"
Nov 02 01:53:45.945844 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:45.945815    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1935b100-31ea-4710-8dad-aa9c4f3ef816-kube-api-access-4fz7r\" (UniqueName: \"kubernetes.io/projected/1935b100-31ea-4710-8dad-aa9c4f3ef816-kube-api-access-4fz7r\") pod \"1935b100-31ea-4710-8dad-aa9c4f3ef816\" (UID: \"1935b100-31ea-4710-8dad-aa9c4f3ef816\") "
Nov 02 01:53:45.945988 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:45.945970    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1935b100-31ea-4710-8dad-aa9c4f3ef816-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1935b100-31ea-4710-8dad-aa9c4f3ef816-plugins-config-volume\") pod \"1935b100-31ea-4710-8dad-aa9c4f3ef816\" (UID: \"1935b100-31ea-4710-8dad-aa9c4f3ef816\") "
Nov 02 01:53:45.946477 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:45.946448    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1935b100-31ea-4710-8dad-aa9c4f3ef816-plugins-config-volume" pod "1935b100-31ea-4710-8dad-aa9c4f3ef816" (UID: "1935b100-31ea-4710-8dad-aa9c4f3ef816"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:53:45.946873 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:45.946850    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1935b100-31ea-4710-8dad-aa9c4f3ef816-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:45.949212 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:45.949187    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1935b100-31ea-4710-8dad-aa9c4f3ef816-kube-api-access-4fz7r" pod "1935b100-31ea-4710-8dad-aa9c4f3ef816" (UID: "1935b100-31ea-4710-8dad-aa9c4f3ef816"). InnerVolumeSpecName "kube-api-access-4fz7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:53:46.047748 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:46.047668    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d-plugins-config-volume\") pod \"80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d\" (UID: \"80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d\") "
Nov 02 01:53:46.047891 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:46.047873    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d-kube-api-access-xzxzs\" (UniqueName: \"kubernetes.io/projected/80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d-kube-api-access-xzxzs\") pod \"80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d\" (UID: \"80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d\") "
Nov 02 01:53:46.048131 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:46.048092    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d-plugins-config-volume" pod "80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d" (UID: "80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:53:46.048627 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:46.048607    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:46.048703 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:46.048634    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fz7r\" (UniqueName: \"kubernetes.io/projected/1935b100-31ea-4710-8dad-aa9c4f3ef816-kube-api-access-4fz7r\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:46.050162 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:46.050141    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d-kube-api-access-xzxzs" pod "80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d" (UID: "80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d"). InnerVolumeSpecName "kube-api-access-xzxzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:53:46.150274 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:46.150241    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzxzs\" (UniqueName: \"kubernetes.io/projected/80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d-kube-api-access-xzxzs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:46.264927 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:46.264888    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d"]
Nov 02 01:53:46.275227 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:46.275198    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8ebd9167-65474c9c57-bvc5d"]
Nov 02 01:53:47.109304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:47.100519    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6" event={"ID":"80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d","Type":"ContainerDied","Data":"9fd8f5129d839fe76444ae0f217b5f4fee478d169e830b859c12204d8b2dc693"}
Nov 02 01:53:47.109304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:47.100669    5639 scope.go:122] "RemoveContainer" containerID="0bcbbe96df9e1f6763ed7d4834078e69e4cb419705e198dab89ca02880d6cc9b"
Nov 02 01:53:47.109304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:47.100908    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6"
Nov 02 01:53:47.227090 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:47.227053    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1935b100-31ea-4710-8dad-aa9c4f3ef816" path="/var/lib/kubelet/pods/1935b100-31ea-4710-8dad-aa9c4f3ef816/volumes"
Nov 02 01:53:47.308530 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:47.308494    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6"]
Nov 02 01:53:47.357479 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:47.357447    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8ebd9167-645ff7c5c5-tzdb6"]
Nov 02 01:53:48.300659 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.300629    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"]
Nov 02 01:53:48.301367 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.301351    5639 cpu_manager.go:414] "containerMap: removing container" podUID="1935b100-31ea-4710-8dad-aa9c4f3ef816" containerName="payload-processing"
Nov 02 01:53:48.301367 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.301366    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1935b100-31ea-4710-8dad-aa9c4f3ef816" containerName="payload-processing"
Nov 02 01:53:48.301481 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.301386    5639 cpu_manager.go:414] "containerMap: removing container" podUID="76863b48-cafa-4e55-adb6-f98efc604043" containerName="revoke-keys"
Nov 02 01:53:48.301481 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.301392    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76863b48-cafa-4e55-adb6-f98efc604043" containerName="revoke-keys"
Nov 02 01:53:48.301481 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.301399    5639 cpu_manager.go:414] "containerMap: removing container" podUID="80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d" containerName="payload-pre-processing"
Nov 02 01:53:48.301481 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.301414    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d" containerName="payload-pre-processing"
Nov 02 01:53:48.301615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.301557    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="1935b100-31ea-4710-8dad-aa9c4f3ef816" containerName="payload-processing"
Nov 02 01:53:48.301615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.301567    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d" containerName="payload-pre-processing"
Nov 02 01:53:48.301615 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.301574    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="76863b48-cafa-4e55-adb6-f98efc604043" containerName="revoke-keys"
Nov 02 01:53:48.306697 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.306672    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.358404 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.354593    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-istio-data\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.358404 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.354761    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-workload-socket\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.358404 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.354801    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70900cc0-155e-46f3-bf31-14169d79b332-istio-token\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.358404 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.354856    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70900cc0-155e-46f3-bf31-14169d79b332-istiod-ca-cert\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.358404 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.355004    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-istio-envoy\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.358404 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.355055    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjb7c\" (UniqueName: \"kubernetes.io/projected/70900cc0-155e-46f3-bf31-14169d79b332-kube-api-access-pjb7c\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.358404 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.355183    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70900cc0-155e-46f3-bf31-14169d79b332-istio-podinfo\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.358404 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.355236    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-credential-socket\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.358404 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.355571    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-workload-certs\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.573507 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.556367    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-257466-openshift-default-dockercfg-j6d2x\""
Nov 02 01:53:48.573507 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.556958    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-workload-socket\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.573507 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.557005    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70900cc0-155e-46f3-bf31-14169d79b332-istio-token\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.573507 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.557084    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70900cc0-155e-46f3-bf31-14169d79b332-istiod-ca-cert\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.573507 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.557245    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-istio-envoy\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.645265 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.607261    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-workload-socket\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.645265 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.607887    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70900cc0-155e-46f3-bf31-14169d79b332-istiod-ca-cert\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.645265 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.630106    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjb7c\" (UniqueName: \"kubernetes.io/projected/70900cc0-155e-46f3-bf31-14169d79b332-kube-api-access-pjb7c\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.645265 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.630259    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70900cc0-155e-46f3-bf31-14169d79b332-istio-podinfo\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.668767 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.668732    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-credential-socket\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.669247 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.669209    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-workload-certs\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.684114 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.684083    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-istio-data\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.684571 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.673029    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-credential-socket\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.684731 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.679914    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70900cc0-155e-46f3-bf31-14169d79b332-istio-podinfo\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.684870 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.669710    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-workload-certs\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.685230 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.685195    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-istio-envoy\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.685635 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.685612    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-istio-data\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.695838 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.695784    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"]
Nov 02 01:53:48.701353 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.701313    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70900cc0-155e-46f3-bf31-14169d79b332-istio-token\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.706933 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.706904    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjb7c\" (UniqueName: \"kubernetes.io/projected/70900cc0-155e-46f3-bf31-14169d79b332-kube-api-access-pjb7c\") pod \"e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") " pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:48.758474 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.758444    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n"
Nov 02 01:53:48.758645 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.758512    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh"
Nov 02 01:53:48.973320 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:48.968570    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:49.229511 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:49.229439    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d" path="/var/lib/kubelet/pods/80a2dfd1-73ae-48e6-b0fd-7d9b8d78751d/volumes"
Nov 02 01:53:49.562602 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:53:49.562572    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70900cc0_155e_46f3_bf31_14169d79b332.slice/crio-cc69585e677bc597ba1fc5ab1539461722a00699266428e75a319d111a0b43bb WatchSource:0}: Error finding container cc69585e677bc597ba1fc5ab1539461722a00699266428e75a319d111a0b43bb: Status 404 returned error can't find the container with id cc69585e677bc597ba1fc5ab1539461722a00699266428e75a319d111a0b43bb
Nov 02 01:53:49.563028 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:49.562881    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"]
Nov 02 01:53:49.565367 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:49.565337    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:53:49.565435 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:49.565398    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:53:49.565435 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:49.565428    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:53:50.164189 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:50.164146    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g" event={"ID":"70900cc0-155e-46f3-bf31-14169d79b332","Type":"ContainerStarted","Data":"8964a3324e240077aca0c73deb70a63366b0077d5e482387449a2cedc0b79ca5"}
Nov 02 01:53:50.164409 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:50.164211    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g" event={"ID":"70900cc0-155e-46f3-bf31-14169d79b332","Type":"ContainerStarted","Data":"cc69585e677bc597ba1fc5ab1539461722a00699266428e75a319d111a0b43bb"}
Nov 02 01:53:50.287621 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:50.287570    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g" podStartSLOduration=2.287555849 podStartE2EDuration="2.287555849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:50.283125517 +0000 UTC m=+1695.186807669" watchObservedRunningTime="2026-11-02 01:53:50.287555849 +0000 UTC m=+1695.191237961"
Nov 02 01:53:50.975047 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:50.975017    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:50.981029 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:50.981002    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:51.172775 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:51.172742    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:51.173646 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:51.173626    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:53:55.050446 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:55.050410    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8151ea22-xkh47"]
Nov 02 01:53:55.096075 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:55.096042    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8151ea22-xkh47"]
Nov 02 01:53:55.100141 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:55.100117    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3a72816-tcd6q"]
Nov 02 01:53:55.104706 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:55.104688    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a72816-tcd6q"
Nov 02 01:53:55.145998 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:55.145969    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b04800a-455b-497e-b3a9-7ee75f8e62f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3a72816-tcd6q\" (UID: \"3b04800a-455b-497e-b3a9-7ee75f8e62f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a72816-tcd6q"
Nov 02 01:53:55.146215 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:55.146192    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b04800a-455b-497e-b3a9-7ee75f8e62f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3a72816-tcd6q\" (UID: \"3b04800a-455b-497e-b3a9-7ee75f8e62f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a72816-tcd6q"
Nov 02 01:53:55.146410 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:55.146394    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmgsn\" (UniqueName: \"kubernetes.io/projected/3b04800a-455b-497e-b3a9-7ee75f8e62f6-kube-api-access-zmgsn\") pod \"maas-api-revoke-sub-a3a72816-tcd6q\" (UID: \"3b04800a-455b-497e-b3a9-7ee75f8e62f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a72816-tcd6q"
Nov 02 01:53:55.162764 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:55.157641    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:53:55.162764 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:55.157676    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:53:55.162764 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:55.162124    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3a72816-tcd6q"]
Nov 02 01:53:55.178652 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:55.178623    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:53:55.178983 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:55.178668    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:53:55.224624 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:55.224587    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2baea30f-f4e8-4c30-9754-0a1192ad0b26" path="/var/lib/kubelet/pods/2baea30f-f4e8-4c30-9754-0a1192ad0b26/volumes"
Nov 02 01:53:55.248148 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:55.248119    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b04800a-455b-497e-b3a9-7ee75f8e62f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3a72816-tcd6q\" (UID: \"3b04800a-455b-497e-b3a9-7ee75f8e62f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a72816-tcd6q"
Nov 02 01:53:55.248373 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:55.248201    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmgsn\" (UniqueName: \"kubernetes.io/projected/3b04800a-455b-497e-b3a9-7ee75f8e62f6-kube-api-access-zmgsn\") pod \"maas-api-revoke-sub-a3a72816-tcd6q\" (UID: \"3b04800a-455b-497e-b3a9-7ee75f8e62f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a72816-tcd6q"
Nov 02 01:53:55.248455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:55.248371    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b04800a-455b-497e-b3a9-7ee75f8e62f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3a72816-tcd6q\" (UID: \"3b04800a-455b-497e-b3a9-7ee75f8e62f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a72816-tcd6q"
Nov 02 01:53:55.248913 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:55.248890    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b04800a-455b-497e-b3a9-7ee75f8e62f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3a72816-tcd6q\" (UID: \"3b04800a-455b-497e-b3a9-7ee75f8e62f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a72816-tcd6q"
Nov 02 01:53:55.251010 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:55.250992    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b04800a-455b-497e-b3a9-7ee75f8e62f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3a72816-tcd6q\" (UID: \"3b04800a-455b-497e-b3a9-7ee75f8e62f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a72816-tcd6q"
Nov 02 01:53:55.308863 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:55.308798    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmgsn\" (UniqueName: \"kubernetes.io/projected/3b04800a-455b-497e-b3a9-7ee75f8e62f6-kube-api-access-zmgsn\") pod \"maas-api-revoke-sub-a3a72816-tcd6q\" (UID: \"3b04800a-455b-497e-b3a9-7ee75f8e62f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a72816-tcd6q"
Nov 02 01:53:55.419928 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:55.419893    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a72816-tcd6q"
Nov 02 01:53:55.804435 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:55.804402    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3a72816-tcd6q"]
Nov 02 01:53:55.804691 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:53:55.804659    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b04800a_455b_497e_b3a9_7ee75f8e62f6.slice/crio-2d1f784244b0f2c332e53b20470191098a047ac5f6dbe6f363e16c4288dca1d4 WatchSource:0}: Error finding container 2d1f784244b0f2c332e53b20470191098a047ac5f6dbe6f363e16c4288dca1d4: Status 404 returned error can't find the container with id 2d1f784244b0f2c332e53b20470191098a047ac5f6dbe6f363e16c4288dca1d4
Nov 02 01:53:56.221004 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:56.220967    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a72816-tcd6q" event={"ID":"3b04800a-455b-497e-b3a9-7ee75f8e62f6","Type":"ContainerStarted","Data":"71da0847bc55babba7a758308b5d1f2348c91ef7ae2bd959f3e5ea460f74db60"}
Nov 02 01:53:56.221384 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:56.221020    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a72816-tcd6q" event={"ID":"3b04800a-455b-497e-b3a9-7ee75f8e62f6","Type":"ContainerStarted","Data":"2d1f784244b0f2c332e53b20470191098a047ac5f6dbe6f363e16c4288dca1d4"}
Nov 02 01:53:56.336235 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:56.335948    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a72816-tcd6q" podStartSLOduration=1.335929508 podStartE2EDuration="1.335929508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:56.335526258 +0000 UTC m=+1701.239208371" watchObservedRunningTime="2026-11-02 01:53:56.335929508 +0000 UTC m=+1701.239611622"
Nov 02 01:53:57.230335 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:57.230268    5639 generic.go:356] "Generic (PLEG): container finished" podID="3b04800a-455b-497e-b3a9-7ee75f8e62f6" containerID="71da0847bc55babba7a758308b5d1f2348c91ef7ae2bd959f3e5ea460f74db60" exitCode=0
Nov 02 01:53:57.230742 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:57.230370    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a72816-tcd6q" event={"ID":"3b04800a-455b-497e-b3a9-7ee75f8e62f6","Type":"ContainerDied","Data":"71da0847bc55babba7a758308b5d1f2348c91ef7ae2bd959f3e5ea460f74db60"}
Nov 02 01:53:57.422242 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:53:57.422207    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:53:58.413961 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:58.413937    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a72816-tcd6q"
Nov 02 01:53:58.500454 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:58.500374    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b04800a-455b-497e-b3a9-7ee75f8e62f6-kube-api-access-zmgsn\" (UniqueName: \"kubernetes.io/projected/3b04800a-455b-497e-b3a9-7ee75f8e62f6-kube-api-access-zmgsn\") pod \"3b04800a-455b-497e-b3a9-7ee75f8e62f6\" (UID: \"3b04800a-455b-497e-b3a9-7ee75f8e62f6\") "
Nov 02 01:53:58.500454 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:58.500444    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b04800a-455b-497e-b3a9-7ee75f8e62f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b04800a-455b-497e-b3a9-7ee75f8e62f6-maas-api-xks-ca\") pod \"3b04800a-455b-497e-b3a9-7ee75f8e62f6\" (UID: \"3b04800a-455b-497e-b3a9-7ee75f8e62f6\") "
Nov 02 01:53:58.500692 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:58.500502    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b04800a-455b-497e-b3a9-7ee75f8e62f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b04800a-455b-497e-b3a9-7ee75f8e62f6-maas-api-openshift-ca\") pod \"3b04800a-455b-497e-b3a9-7ee75f8e62f6\" (UID: \"3b04800a-455b-497e-b3a9-7ee75f8e62f6\") "
Nov 02 01:53:58.500933 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:58.500901    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b04800a-455b-497e-b3a9-7ee75f8e62f6-maas-api-openshift-ca" pod "3b04800a-455b-497e-b3a9-7ee75f8e62f6" (UID: "3b04800a-455b-497e-b3a9-7ee75f8e62f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:53:58.501245 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:58.501225    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b04800a-455b-497e-b3a9-7ee75f8e62f6-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:58.502729 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:58.502696    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b04800a-455b-497e-b3a9-7ee75f8e62f6-maas-api-xks-ca" pod "3b04800a-455b-497e-b3a9-7ee75f8e62f6" (UID: "3b04800a-455b-497e-b3a9-7ee75f8e62f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:53:58.502810 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:58.502725    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b04800a-455b-497e-b3a9-7ee75f8e62f6-kube-api-access-zmgsn" pod "3b04800a-455b-497e-b3a9-7ee75f8e62f6" (UID: "3b04800a-455b-497e-b3a9-7ee75f8e62f6"). InnerVolumeSpecName "kube-api-access-zmgsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:53:58.602877 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:58.602838    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmgsn\" (UniqueName: \"kubernetes.io/projected/3b04800a-455b-497e-b3a9-7ee75f8e62f6-kube-api-access-zmgsn\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:58.602877 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:58.602879    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b04800a-455b-497e-b3a9-7ee75f8e62f6-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:59.250177 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:59.250150    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a72816-tcd6q"
Nov 02 01:53:59.250417 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:59.250176    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a72816-tcd6q" event={"ID":"3b04800a-455b-497e-b3a9-7ee75f8e62f6","Type":"ContainerDied","Data":"2d1f784244b0f2c332e53b20470191098a047ac5f6dbe6f363e16c4288dca1d4"}
Nov 02 01:53:59.250417 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:53:59.250232    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d1f784244b0f2c332e53b20470191098a047ac5f6dbe6f363e16c4288dca1d4"
Nov 02 01:54:06.066057 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:06.066020    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:54:09.069100 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:09.069058    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a330ba00-whkqc"]
Nov 02 01:54:09.156597 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:09.156554    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a330ba00-whkqc"]
Nov 02 01:54:09.366313 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:09.331902    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26bb5d7f-26f6-4a7f-9dba-e5b3239b536c" path="/var/lib/kubelet/pods/26bb5d7f-26f6-4a7f-9dba-e5b3239b536c/volumes"
Nov 02 01:54:09.472477 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:09.472434    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:09.939306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:09.920813    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:09.939306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:09.921151    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:09.939306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:09.921178    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Nov 02 01:54:09.939306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:09.922270    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:09.939306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:09.922359    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:09.976852 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:09.976809    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:10.145402 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:10.145354    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:54:10.162507 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:10.162444    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 02 01:54:10.363806 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:10.363762    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:10.475424 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:10.475378    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:54:10.476309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:10.476269    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:54:10.755739 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:10.755582    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:54:16.920302 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:16.919557    5639 patch_prober.go:29] interesting pod/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:54:16.920302 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:16.919662    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d" podUID="86b56f9b-4c16-412c-b170-76133ab7abb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:19.447976 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:19.447916    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd"]
Nov 02 01:54:19.495365 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:19.495331    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ebd9167-pk4fd"]
Nov 02 01:54:19.506399 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:19.506318    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"]
Nov 02 01:54:19.506734 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:19.506671    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb" podUID="d8ba9d0e-66ce-4927-aed2-e84b3280e116" containerName="istio-proxy" containerID="cri-o://d56a3ead53902f02de26e785569ad9adfd55f6517c54a1ddc8213f86c19f163e" gracePeriod=30
Nov 02 01:54:21.410340 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:21.409951    5639 generic.go:356] "Generic (PLEG): container finished" podID="86b56f9b-4c16-412c-b170-76133ab7abb2" containerID="fbeb990c66f31886a87770784922ccdee559e57d9929090671b9a8fb659e3375" exitCode=0
Nov 02 01:54:21.580841 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:21.526246    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758" path="/var/lib/kubelet/pods/4e5bff52-8fe8-4bf6-9aa3-c0578ce9b758/volumes"
Nov 02 01:54:21.580841 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:21.573759    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d" event={"ID":"86b56f9b-4c16-412c-b170-76133ab7abb2","Type":"ContainerDied","Data":"fbeb990c66f31886a87770784922ccdee559e57d9929090671b9a8fb659e3375"}
Nov 02 01:54:21.580841 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:21.575862    5639 scope.go:122] "RemoveContainer" containerID="fbeb990c66f31886a87770784922ccdee559e57d9929090671b9a8fb659e3375"
Nov 02 01:54:21.613304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:21.607608    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:54:21.613304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:21.607734    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:54:21.613304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:21.607783    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:54:22.421390 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:22.421347    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d" event={"ID":"86b56f9b-4c16-412c-b170-76133ab7abb2","Type":"ContainerStarted","Data":"561ab8df5652445fbd5eb4a2abb2df89dfcba47c475976070e4a24ce46ecd4cd"}
Nov 02 01:54:22.941311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:22.937716    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:54:22.941311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:22.940106    5639 patch_prober.go:29] interesting pod/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Nov 02 01:54:22.941311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:22.940179    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d" podUID="86b56f9b-4c16-412c-b170-76133ab7abb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Nov 02 01:54:23.083021 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:23.082981    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f154566-b8n47"]
Nov 02 01:54:23.106028 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:23.105977    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f154566-b8n47"]
Nov 02 01:54:23.222129 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:23.222046    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="080f5dc7-c78f-4c7f-ab59-e7e05988b487" path="/var/lib/kubelet/pods/080f5dc7-c78f-4c7f-ab59-e7e05988b487/volumes"
Nov 02 01:54:23.940422 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:23.939146    5639 patch_prober.go:29] interesting pod/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Nov 02 01:54:23.940422 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:23.939207    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d" podUID="86b56f9b-4c16-412c-b170-76133ab7abb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Nov 02 01:54:24.935260 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:24.935184    5639 patch_prober.go:29] interesting pod/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Nov 02 01:54:24.935260 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:24.935247    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d" podUID="86b56f9b-4c16-412c-b170-76133ab7abb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Nov 02 01:54:25.111096 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.111067    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:54:25.200522 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.200443    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-credential-socket\") pod \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") "
Nov 02 01:54:25.200522 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.200477    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-podinfo\") pod \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") "
Nov 02 01:54:25.200522 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.200514    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-workload-certs\") pod \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") "
Nov 02 01:54:25.200797 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.200595    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-token\" (UniqueName: \"kubernetes.io/projected/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-token\") pod \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") "
Nov 02 01:54:25.200797 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.200662    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-envoy\") pod \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") "
Nov 02 01:54:25.200797 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.200755    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-workload-certs" pod "d8ba9d0e-66ce-4927-aed2-e84b3280e116" (UID: "d8ba9d0e-66ce-4927-aed2-e84b3280e116"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:25.200962 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.200820    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istiod-ca-cert\") pod \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") "
Nov 02 01:54:25.200962 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.200835    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-credential-socket" pod "d8ba9d0e-66ce-4927-aed2-e84b3280e116" (UID: "d8ba9d0e-66ce-4927-aed2-e84b3280e116"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:25.201080 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.200960    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-data\") pod \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") "
Nov 02 01:54:25.201080 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.200997    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-workload-socket\") pod \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") "
Nov 02 01:54:25.201080 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.201063    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8ba9d0e-66ce-4927-aed2-e84b3280e116-kube-api-access-7m52q\" (UniqueName: \"kubernetes.io/projected/d8ba9d0e-66ce-4927-aed2-e84b3280e116-kube-api-access-7m52q\") pod \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\" (UID: \"d8ba9d0e-66ce-4927-aed2-e84b3280e116\") "
Nov 02 01:54:25.201256 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.201229    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-data" pod "d8ba9d0e-66ce-4927-aed2-e84b3280e116" (UID: "d8ba9d0e-66ce-4927-aed2-e84b3280e116"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:25.201373 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.201314    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istiod-ca-cert" pod "d8ba9d0e-66ce-4927-aed2-e84b3280e116" (UID: "d8ba9d0e-66ce-4927-aed2-e84b3280e116"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:25.201524 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.201482    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-workload-socket" pod "d8ba9d0e-66ce-4927-aed2-e84b3280e116" (UID: "d8ba9d0e-66ce-4927-aed2-e84b3280e116"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:25.202094 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.202072    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:25.202094 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.202095    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:25.202263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.202104    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:25.202263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.202112    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:25.202263 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.202123    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:25.203016 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.202993    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-envoy" pod "d8ba9d0e-66ce-4927-aed2-e84b3280e116" (UID: "d8ba9d0e-66ce-4927-aed2-e84b3280e116"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:25.203636 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.203606    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-podinfo" pod "d8ba9d0e-66ce-4927-aed2-e84b3280e116" (UID: "d8ba9d0e-66ce-4927-aed2-e84b3280e116"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:54:25.203742 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.203644    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ba9d0e-66ce-4927-aed2-e84b3280e116-kube-api-access-7m52q" pod "d8ba9d0e-66ce-4927-aed2-e84b3280e116" (UID: "d8ba9d0e-66ce-4927-aed2-e84b3280e116"). InnerVolumeSpecName "kube-api-access-7m52q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:25.204052 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.204033    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-token" pod "d8ba9d0e-66ce-4927-aed2-e84b3280e116" (UID: "d8ba9d0e-66ce-4927-aed2-e84b3280e116"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:25.303934 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.303894    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m52q\" (UniqueName: \"kubernetes.io/projected/d8ba9d0e-66ce-4927-aed2-e84b3280e116-kube-api-access-7m52q\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:25.303934 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.303931    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:25.304186 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.303945    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:25.304186 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.303955    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ba9d0e-66ce-4927-aed2-e84b3280e116-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:25.450416 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.450381    5639 generic.go:356] "Generic (PLEG): container finished" podID="d8ba9d0e-66ce-4927-aed2-e84b3280e116" containerID="d56a3ead53902f02de26e785569ad9adfd55f6517c54a1ddc8213f86c19f163e" exitCode=0
Nov 02 01:54:25.450624 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.450449    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"
Nov 02 01:54:25.450624 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.450459    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb" event={"ID":"d8ba9d0e-66ce-4927-aed2-e84b3280e116","Type":"ContainerDied","Data":"d56a3ead53902f02de26e785569ad9adfd55f6517c54a1ddc8213f86c19f163e"}
Nov 02 01:54:25.450624 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.450520    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb" event={"ID":"d8ba9d0e-66ce-4927-aed2-e84b3280e116","Type":"ContainerDied","Data":"df5bfeab3f9ab0da0399f54bf56a0e2cb68241e60fd154b3450e7d03e3b64a68"}
Nov 02 01:54:25.450624 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.450550    5639 scope.go:122] "RemoveContainer" containerID="d56a3ead53902f02de26e785569ad9adfd55f6517c54a1ddc8213f86c19f163e"
Nov 02 01:54:25.486568 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.486545    5639 scope.go:122] "RemoveContainer" containerID="0a1c65adade57ed2ad17262856d7e3f5da351049750421a7b5efd8f0bdb10f26"
Nov 02 01:54:25.500233 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.500208    5639 scope.go:122] "RemoveContainer" containerID="d56a3ead53902f02de26e785569ad9adfd55f6517c54a1ddc8213f86c19f163e"
Nov 02 01:54:25.500560 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:54:25.500537    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d56a3ead53902f02de26e785569ad9adfd55f6517c54a1ddc8213f86c19f163e\": container with ID starting with d56a3ead53902f02de26e785569ad9adfd55f6517c54a1ddc8213f86c19f163e not found: ID does not exist" containerID="d56a3ead53902f02de26e785569ad9adfd55f6517c54a1ddc8213f86c19f163e"
Nov 02 01:54:25.500671 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.500569    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d56a3ead53902f02de26e785569ad9adfd55f6517c54a1ddc8213f86c19f163e"} err="failed to get container status \"d56a3ead53902f02de26e785569ad9adfd55f6517c54a1ddc8213f86c19f163e\": rpc error: code = NotFound desc = could not find container \"d56a3ead53902f02de26e785569ad9adfd55f6517c54a1ddc8213f86c19f163e\": container with ID starting with d56a3ead53902f02de26e785569ad9adfd55f6517c54a1ddc8213f86c19f163e not found: ID does not exist"
Nov 02 01:54:25.500671 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.500591    5639 scope.go:122] "RemoveContainer" containerID="0a1c65adade57ed2ad17262856d7e3f5da351049750421a7b5efd8f0bdb10f26"
Nov 02 01:54:25.500875 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:54:25.500848    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a1c65adade57ed2ad17262856d7e3f5da351049750421a7b5efd8f0bdb10f26\": container with ID starting with 0a1c65adade57ed2ad17262856d7e3f5da351049750421a7b5efd8f0bdb10f26 not found: ID does not exist" containerID="0a1c65adade57ed2ad17262856d7e3f5da351049750421a7b5efd8f0bdb10f26"
Nov 02 01:54:25.500915 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.500881    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a1c65adade57ed2ad17262856d7e3f5da351049750421a7b5efd8f0bdb10f26"} err="failed to get container status \"0a1c65adade57ed2ad17262856d7e3f5da351049750421a7b5efd8f0bdb10f26\": rpc error: code = NotFound desc = could not find container \"0a1c65adade57ed2ad17262856d7e3f5da351049750421a7b5efd8f0bdb10f26\": container with ID starting with 0a1c65adade57ed2ad17262856d7e3f5da351049750421a7b5efd8f0bdb10f26 not found: ID does not exist"
Nov 02 01:54:25.625875 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.625840    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"]
Nov 02 01:54:25.630133 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.630109    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-8ebd9167-gw-openshift-default-5979c6f88f-b8nkb"]
Nov 02 01:54:25.935848 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.935814    5639 patch_prober.go:29] interesting pod/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Nov 02 01:54:25.936029 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:25.935863    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d" podUID="86b56f9b-4c16-412c-b170-76133ab7abb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Nov 02 01:54:26.935734 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:26.935701    5639 patch_prober.go:29] interesting pod/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Nov 02 01:54:26.936217 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:26.935771    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d" podUID="86b56f9b-4c16-412c-b170-76133ab7abb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Nov 02 01:54:27.220223 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:27.220140    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8ba9d0e-66ce-4927-aed2-e84b3280e116" path="/var/lib/kubelet/pods/d8ba9d0e-66ce-4927-aed2-e84b3280e116/volumes"
Nov 02 01:54:27.420375 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:27.420341    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"]
Nov 02 01:54:27.420620 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:27.420589    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc" podUID="e5ab0dca-9dde-45ae-8647-61a346c461f2" containerName="istio-proxy" containerID="cri-o://746680d693e3d8728af7b852f710d2a687337ab69215189cfd91fe21c0b1c74e" gracePeriod=30
Nov 02 01:54:27.424853 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:54:27.424817    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:54:27.488330 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:27.488204    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"]
Nov 02 01:54:27.488530 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:27.488433    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8" podUID="3da5e6df-b15c-4209-9cec-624599f6713d" containerName="istio-proxy" containerID="cri-o://3a010c030ef32387455a5a37809d0239a3978b475bf62b6554cbfffa65afd7c9" gracePeriod=30
Nov 02 01:54:27.494767 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:27.494737    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"]
Nov 02 01:54:27.495018 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:27.494985    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb" podUID="fb887678-0155-4d81-be15-861a7e1f8ab8" containerName="istio-proxy" containerID="cri-o://53cbe30adf15019a6faf0f6ae85b07888acec8d05bbaa9964ffb1112870ddb2e" gracePeriod=30
Nov 02 01:54:27.940908 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:27.940849    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:54:27.941433 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:27.941416    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:54:27.942059 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:27.942040    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:54:28.685324 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:28.685258    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"]
Nov 02 01:54:28.685575 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:28.685535    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"]
Nov 02 01:54:28.685891 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:28.685833    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz" podUID="1f799283-3131-46a5-b010-628029210780" containerName="istio-proxy" containerID="cri-o://19b8c99f9f3e6a460f55fbc5880b434a658cea21501e8c09be3c18e9d42e1b2a" gracePeriod=30
Nov 02 01:54:28.686084 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:28.686040    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp" podUID="945be3db-df2f-4ef9-9609-a7890bee84ce" containerName="istio-proxy" containerID="cri-o://e07eb0ed26cff5e32d54db412e9bea74ff8f87d80a1aceed083e11651b35168e" gracePeriod=30
Nov 02 01:54:28.726436 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:28.726405    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"]
Nov 02 01:54:28.726677 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:28.726626    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5" podUID="9984c750-63ba-4996-a4b1-44549030fe24" containerName="istio-proxy" containerID="cri-o://f62dd4fc2e17f5b9b6c749612001a52f659c6a2f5f9fc324725bdab67d48a4e6" gracePeriod=30
Nov 02 01:54:28.731968 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:28.731942    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"]
Nov 02 01:54:28.732178 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:28.732139    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h" podUID="1640ee78-dc59-4afb-a7f6-e130f60cdf4a" containerName="istio-proxy" containerID="cri-o://9ac49c154511182455be13f90b691f0238fa39d2a62e516a02dd8e425f2e0a64" gracePeriod=30
Nov 02 01:54:32.262714 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.262675    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:54:32.263102 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.262750    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h" podUID="1640ee78-dc59-4afb-a7f6-e130f60cdf4a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:54:32.466811 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.466777    5639 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:54:32.466965 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.466853    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp" podUID="945be3db-df2f-4ef9-9609-a7890bee84ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:54:32.554517 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.554482    5639 generic.go:356] "Generic (PLEG): container finished" podID="e5ab0dca-9dde-45ae-8647-61a346c461f2" containerID="746680d693e3d8728af7b852f710d2a687337ab69215189cfd91fe21c0b1c74e" exitCode=0
Nov 02 01:54:32.554684 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.554520    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc" event={"ID":"e5ab0dca-9dde-45ae-8647-61a346c461f2","Type":"ContainerDied","Data":"746680d693e3d8728af7b852f710d2a687337ab69215189cfd91fe21c0b1c74e"}
Nov 02 01:54:32.792374 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.792265    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:54:32.824879 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.824846    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:54:32.876367 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.876336    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:54:32.913086 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.913047    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-workload-socket\") pod \"fb887678-0155-4d81-be15-861a7e1f8ab8\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") "
Nov 02 01:54:32.913323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.913110    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3da5e6df-b15c-4209-9cec-624599f6713d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3da5e6df-b15c-4209-9cec-624599f6713d-istiod-ca-cert\") pod \"3da5e6df-b15c-4209-9cec-624599f6713d\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") "
Nov 02 01:54:32.913323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.913159    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-workload-socket\") pod \"3da5e6df-b15c-4209-9cec-624599f6713d\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") "
Nov 02 01:54:32.913323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.913193    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-credential-socket\") pod \"3da5e6df-b15c-4209-9cec-624599f6713d\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") "
Nov 02 01:54:32.913323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.913260    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3da5e6df-b15c-4209-9cec-624599f6713d-kube-api-access-nxdp5\" (UniqueName: \"kubernetes.io/projected/3da5e6df-b15c-4209-9cec-624599f6713d-kube-api-access-nxdp5\") pod \"3da5e6df-b15c-4209-9cec-624599f6713d\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") "
Nov 02 01:54:32.913567 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.913335    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-envoy\") pod \"fb887678-0155-4d81-be15-861a7e1f8ab8\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") "
Nov 02 01:54:32.913567 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.913369    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-workload-socket\") pod \"e5ab0dca-9dde-45ae-8647-61a346c461f2\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") "
Nov 02 01:54:32.913567 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.913415    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-workload-certs\") pod \"fb887678-0155-4d81-be15-861a7e1f8ab8\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") "
Nov 02 01:54:32.913567 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.913456    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-workload-certs\") pod \"e5ab0dca-9dde-45ae-8647-61a346c461f2\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") "
Nov 02 01:54:32.913567 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.913497    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-credential-socket\") pod \"fb887678-0155-4d81-be15-861a7e1f8ab8\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") "
Nov 02 01:54:32.913829 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.913575    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-token\" (UniqueName: \"kubernetes.io/projected/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-token\") pod \"fb887678-0155-4d81-be15-861a7e1f8ab8\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") "
Nov 02 01:54:32.913829 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.913611    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-data\") pod \"e5ab0dca-9dde-45ae-8647-61a346c461f2\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") "
Nov 02 01:54:32.913829 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.913675    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb887678-0155-4d81-be15-861a7e1f8ab8-kube-api-access-dxnqk\" (UniqueName: \"kubernetes.io/projected/fb887678-0155-4d81-be15-861a7e1f8ab8-kube-api-access-dxnqk\") pod \"fb887678-0155-4d81-be15-861a7e1f8ab8\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") "
Nov 02 01:54:32.913829 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.913717    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-credential-socket\") pod \"e5ab0dca-9dde-45ae-8647-61a346c461f2\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") "
Nov 02 01:54:32.913829 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.913743    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-token\" (UniqueName: \"kubernetes.io/projected/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-token\") pod \"e5ab0dca-9dde-45ae-8647-61a346c461f2\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") "
Nov 02 01:54:32.913829 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.913787    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-envoy\") pod \"e5ab0dca-9dde-45ae-8647-61a346c461f2\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") "
Nov 02 01:54:32.914117 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.913837    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb887678-0155-4d81-be15-861a7e1f8ab8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb887678-0155-4d81-be15-861a7e1f8ab8-istiod-ca-cert\") pod \"fb887678-0155-4d81-be15-861a7e1f8ab8\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") "
Nov 02 01:54:32.914117 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.913898    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-workload-certs\") pod \"3da5e6df-b15c-4209-9cec-624599f6713d\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") "
Nov 02 01:54:32.914117 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.913950    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3da5e6df-b15c-4209-9cec-624599f6713d-istio-token\" (UniqueName: \"kubernetes.io/projected/3da5e6df-b15c-4209-9cec-624599f6713d-istio-token\") pod \"3da5e6df-b15c-4209-9cec-624599f6713d\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") "
Nov 02 01:54:32.914117 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.914040    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3da5e6df-b15c-4209-9cec-624599f6713d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3da5e6df-b15c-4209-9cec-624599f6713d-istio-podinfo\") pod \"3da5e6df-b15c-4209-9cec-624599f6713d\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") "
Nov 02 01:54:32.914117 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.914072    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-podinfo\") pod \"e5ab0dca-9dde-45ae-8647-61a346c461f2\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") "
Nov 02 01:54:32.914117 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.914103    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-podinfo\") pod \"fb887678-0155-4d81-be15-861a7e1f8ab8\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") "
Nov 02 01:54:32.914459 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.914137    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-istio-data\") pod \"3da5e6df-b15c-4209-9cec-624599f6713d\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") "
Nov 02 01:54:32.914459 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.914164    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5ab0dca-9dde-45ae-8647-61a346c461f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ab0dca-9dde-45ae-8647-61a346c461f2-istiod-ca-cert\") pod \"e5ab0dca-9dde-45ae-8647-61a346c461f2\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") "
Nov 02 01:54:32.914459 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.914243    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-data\") pod \"fb887678-0155-4d81-be15-861a7e1f8ab8\" (UID: \"fb887678-0155-4d81-be15-861a7e1f8ab8\") "
Nov 02 01:54:32.914459 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.914268    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-istio-envoy\") pod \"3da5e6df-b15c-4209-9cec-624599f6713d\" (UID: \"3da5e6df-b15c-4209-9cec-624599f6713d\") "
Nov 02 01:54:32.914459 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.914383    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ab0dca-9dde-45ae-8647-61a346c461f2-kube-api-access-84hgc\" (UniqueName: \"kubernetes.io/projected/e5ab0dca-9dde-45ae-8647-61a346c461f2-kube-api-access-84hgc\") pod \"e5ab0dca-9dde-45ae-8647-61a346c461f2\" (UID: \"e5ab0dca-9dde-45ae-8647-61a346c461f2\") "
Nov 02 01:54:32.915801 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.915760    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-workload-socket" pod "e5ab0dca-9dde-45ae-8647-61a346c461f2" (UID: "e5ab0dca-9dde-45ae-8647-61a346c461f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:32.915992 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.915967    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-workload-certs" pod "fb887678-0155-4d81-be15-861a7e1f8ab8" (UID: "fb887678-0155-4d81-be15-861a7e1f8ab8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:32.916175 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.916150    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-envoy" pod "fb887678-0155-4d81-be15-861a7e1f8ab8" (UID: "fb887678-0155-4d81-be15-861a7e1f8ab8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:32.916175 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.916168    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-workload-certs" pod "e5ab0dca-9dde-45ae-8647-61a346c461f2" (UID: "e5ab0dca-9dde-45ae-8647-61a346c461f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:32.916394 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.916377    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-credential-socket" pod "fb887678-0155-4d81-be15-861a7e1f8ab8" (UID: "fb887678-0155-4d81-be15-861a7e1f8ab8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:32.916516 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.916493    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-workload-socket" pod "3da5e6df-b15c-4209-9cec-624599f6713d" (UID: "3da5e6df-b15c-4209-9cec-624599f6713d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:32.916698 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.916682    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-credential-socket" pod "3da5e6df-b15c-4209-9cec-624599f6713d" (UID: "3da5e6df-b15c-4209-9cec-624599f6713d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:32.917465 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.917441    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3da5e6df-b15c-4209-9cec-624599f6713d-istiod-ca-cert" pod "3da5e6df-b15c-4209-9cec-624599f6713d" (UID: "3da5e6df-b15c-4209-9cec-624599f6713d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:32.918786 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.918754    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-workload-socket" pod "fb887678-0155-4d81-be15-861a7e1f8ab8" (UID: "fb887678-0155-4d81-be15-861a7e1f8ab8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:32.918911 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.918852    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-token" pod "fb887678-0155-4d81-be15-861a7e1f8ab8" (UID: "fb887678-0155-4d81-be15-861a7e1f8ab8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:32.919092 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.919073    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-data" pod "e5ab0dca-9dde-45ae-8647-61a346c461f2" (UID: "e5ab0dca-9dde-45ae-8647-61a346c461f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:32.919148 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.919099    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da5e6df-b15c-4209-9cec-624599f6713d-kube-api-access-nxdp5" pod "3da5e6df-b15c-4209-9cec-624599f6713d" (UID: "3da5e6df-b15c-4209-9cec-624599f6713d"). InnerVolumeSpecName "kube-api-access-nxdp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:32.919728 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.919478    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-data" pod "fb887678-0155-4d81-be15-861a7e1f8ab8" (UID: "fb887678-0155-4d81-be15-861a7e1f8ab8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:32.925440 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.923293    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb887678-0155-4d81-be15-861a7e1f8ab8-kube-api-access-dxnqk" pod "fb887678-0155-4d81-be15-861a7e1f8ab8" (UID: "fb887678-0155-4d81-be15-861a7e1f8ab8"). InnerVolumeSpecName "kube-api-access-dxnqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:32.925440 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.924085    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-istio-data" pod "3da5e6df-b15c-4209-9cec-624599f6713d" (UID: "3da5e6df-b15c-4209-9cec-624599f6713d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:32.925440 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.924090    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-podinfo" pod "e5ab0dca-9dde-45ae-8647-61a346c461f2" (UID: "e5ab0dca-9dde-45ae-8647-61a346c461f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:54:32.925440 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.924307    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-workload-certs" pod "3da5e6df-b15c-4209-9cec-624599f6713d" (UID: "3da5e6df-b15c-4209-9cec-624599f6713d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:32.925440 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.924483    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb887678-0155-4d81-be15-861a7e1f8ab8-istiod-ca-cert" pod "fb887678-0155-4d81-be15-861a7e1f8ab8" (UID: "fb887678-0155-4d81-be15-861a7e1f8ab8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:32.931392 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.930738    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-envoy" pod "e5ab0dca-9dde-45ae-8647-61a346c461f2" (UID: "e5ab0dca-9dde-45ae-8647-61a346c461f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:32.931392 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.931020    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-token" pod "e5ab0dca-9dde-45ae-8647-61a346c461f2" (UID: "e5ab0dca-9dde-45ae-8647-61a346c461f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:32.931392 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.931256    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-credential-socket" pod "e5ab0dca-9dde-45ae-8647-61a346c461f2" (UID: "e5ab0dca-9dde-45ae-8647-61a346c461f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:32.931661 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.931462    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ab0dca-9dde-45ae-8647-61a346c461f2-istiod-ca-cert" pod "e5ab0dca-9dde-45ae-8647-61a346c461f2" (UID: "e5ab0dca-9dde-45ae-8647-61a346c461f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:32.932407 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.931822    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-podinfo" pod "fb887678-0155-4d81-be15-861a7e1f8ab8" (UID: "fb887678-0155-4d81-be15-861a7e1f8ab8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:54:32.932407 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.932097    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ab0dca-9dde-45ae-8647-61a346c461f2-kube-api-access-84hgc" pod "e5ab0dca-9dde-45ae-8647-61a346c461f2" (UID: "e5ab0dca-9dde-45ae-8647-61a346c461f2"). InnerVolumeSpecName "kube-api-access-84hgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:32.932407 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.932318    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3da5e6df-b15c-4209-9cec-624599f6713d-istio-podinfo" pod "3da5e6df-b15c-4209-9cec-624599f6713d" (UID: "3da5e6df-b15c-4209-9cec-624599f6713d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:54:32.932407 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.932367    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-istio-envoy" pod "3da5e6df-b15c-4209-9cec-624599f6713d" (UID: "3da5e6df-b15c-4209-9cec-624599f6713d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:32.932764 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:32.932738    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da5e6df-b15c-4209-9cec-624599f6713d-istio-token" pod "3da5e6df-b15c-4209-9cec-624599f6713d" (UID: "3da5e6df-b15c-4209-9cec-624599f6713d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:33.016625 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016586    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3da5e6df-b15c-4209-9cec-624599f6713d-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.016625 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016619    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3da5e6df-b15c-4209-9cec-624599f6713d-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.016625 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016630    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.016881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016639    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.016881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016649    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.016881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016658    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ab0dca-9dde-45ae-8647-61a346c461f2-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.016881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016666    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.016881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016674    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.016881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016682    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84hgc\" (UniqueName: \"kubernetes.io/projected/e5ab0dca-9dde-45ae-8647-61a346c461f2-kube-api-access-84hgc\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.016881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016692    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.016881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016700    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3da5e6df-b15c-4209-9cec-624599f6713d-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.016881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016708    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.016881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016716    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.016881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016724    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxdp5\" (UniqueName: \"kubernetes.io/projected/3da5e6df-b15c-4209-9cec-624599f6713d-kube-api-access-nxdp5\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.016881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016733    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.016881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016740    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.016881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016748    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.016881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016757    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.016881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016765    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb887678-0155-4d81-be15-861a7e1f8ab8-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.016881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016772    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb887678-0155-4d81-be15-861a7e1f8ab8-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.016881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016780    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.016881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016788    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxnqk\" (UniqueName: \"kubernetes.io/projected/fb887678-0155-4d81-be15-861a7e1f8ab8-kube-api-access-dxnqk\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.016881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016796    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.016881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016804    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.016881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016811    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ab0dca-9dde-45ae-8647-61a346c461f2-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.016881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016820    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb887678-0155-4d81-be15-861a7e1f8ab8-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.016881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.016831    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3da5e6df-b15c-4209-9cec-624599f6713d-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.565098 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.565006    5639 generic.go:356] "Generic (PLEG): container finished" podID="fb887678-0155-4d81-be15-861a7e1f8ab8" containerID="53cbe30adf15019a6faf0f6ae85b07888acec8d05bbaa9964ffb1112870ddb2e" exitCode=0
Nov 02 01:54:33.565098 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.565069    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"
Nov 02 01:54:33.565630 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.565097    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb" event={"ID":"fb887678-0155-4d81-be15-861a7e1f8ab8","Type":"ContainerDied","Data":"53cbe30adf15019a6faf0f6ae85b07888acec8d05bbaa9964ffb1112870ddb2e"}
Nov 02 01:54:33.565630 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.565183    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb" event={"ID":"fb887678-0155-4d81-be15-861a7e1f8ab8","Type":"ContainerDied","Data":"ef0be024684bbd591431df4b605b914e5a31f5b088bfc72cebb918f93bb57518"}
Nov 02 01:54:33.565630 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.565208    5639 scope.go:122] "RemoveContainer" containerID="53cbe30adf15019a6faf0f6ae85b07888acec8d05bbaa9964ffb1112870ddb2e"
Nov 02 01:54:33.567030 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.567007    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"
Nov 02 01:54:33.567165 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.567006    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc" event={"ID":"e5ab0dca-9dde-45ae-8647-61a346c461f2","Type":"ContainerDied","Data":"aebc71c0afb162a06de2dc24da1586d98bcf2fa7625c3a86c0ffea03cb25fd7d"}
Nov 02 01:54:33.568859 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.568830    5639 generic.go:356] "Generic (PLEG): container finished" podID="3da5e6df-b15c-4209-9cec-624599f6713d" containerID="3a010c030ef32387455a5a37809d0239a3978b475bf62b6554cbfffa65afd7c9" exitCode=0
Nov 02 01:54:33.568983 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.568887    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"
Nov 02 01:54:33.568983 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.568909    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8" event={"ID":"3da5e6df-b15c-4209-9cec-624599f6713d","Type":"ContainerDied","Data":"3a010c030ef32387455a5a37809d0239a3978b475bf62b6554cbfffa65afd7c9"}
Nov 02 01:54:33.568983 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.568960    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8" event={"ID":"3da5e6df-b15c-4209-9cec-624599f6713d","Type":"ContainerDied","Data":"7be4184ab26a1ae9563df26d4a1405a5a594551091a2ae84681b817b96fae0c3"}
Nov 02 01:54:33.578438 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.578416    5639 scope.go:122] "RemoveContainer" containerID="53cbe30adf15019a6faf0f6ae85b07888acec8d05bbaa9964ffb1112870ddb2e"
Nov 02 01:54:33.578727 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:54:33.578709    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53cbe30adf15019a6faf0f6ae85b07888acec8d05bbaa9964ffb1112870ddb2e\": container with ID starting with 53cbe30adf15019a6faf0f6ae85b07888acec8d05bbaa9964ffb1112870ddb2e not found: ID does not exist" containerID="53cbe30adf15019a6faf0f6ae85b07888acec8d05bbaa9964ffb1112870ddb2e"
Nov 02 01:54:33.578783 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.578735    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53cbe30adf15019a6faf0f6ae85b07888acec8d05bbaa9964ffb1112870ddb2e"} err="failed to get container status \"53cbe30adf15019a6faf0f6ae85b07888acec8d05bbaa9964ffb1112870ddb2e\": rpc error: code = NotFound desc = could not find container \"53cbe30adf15019a6faf0f6ae85b07888acec8d05bbaa9964ffb1112870ddb2e\": container with ID starting with 53cbe30adf15019a6faf0f6ae85b07888acec8d05bbaa9964ffb1112870ddb2e not found: ID does not exist"
Nov 02 01:54:33.578783 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.578752    5639 scope.go:122] "RemoveContainer" containerID="746680d693e3d8728af7b852f710d2a687337ab69215189cfd91fe21c0b1c74e"
Nov 02 01:54:33.590028 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.590004    5639 scope.go:122] "RemoveContainer" containerID="3a010c030ef32387455a5a37809d0239a3978b475bf62b6554cbfffa65afd7c9"
Nov 02 01:54:33.602098 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.601925    5639 scope.go:122] "RemoveContainer" containerID="3a010c030ef32387455a5a37809d0239a3978b475bf62b6554cbfffa65afd7c9"
Nov 02 01:54:33.602359 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:54:33.602334    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a010c030ef32387455a5a37809d0239a3978b475bf62b6554cbfffa65afd7c9\": container with ID starting with 3a010c030ef32387455a5a37809d0239a3978b475bf62b6554cbfffa65afd7c9 not found: ID does not exist" containerID="3a010c030ef32387455a5a37809d0239a3978b475bf62b6554cbfffa65afd7c9"
Nov 02 01:54:33.602442 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.602368    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a010c030ef32387455a5a37809d0239a3978b475bf62b6554cbfffa65afd7c9"} err="failed to get container status \"3a010c030ef32387455a5a37809d0239a3978b475bf62b6554cbfffa65afd7c9\": rpc error: code = NotFound desc = could not find container \"3a010c030ef32387455a5a37809d0239a3978b475bf62b6554cbfffa65afd7c9\": container with ID starting with 3a010c030ef32387455a5a37809d0239a3978b475bf62b6554cbfffa65afd7c9 not found: ID does not exist"
Nov 02 01:54:33.732133 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.732107    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"]
Nov 02 01:54:33.738174 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.738144    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-4hzrc"]
Nov 02 01:54:33.854387 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.854355    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"]
Nov 02 01:54:33.860233 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.860206    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-9pdcb"]
Nov 02 01:54:33.974141 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.974055    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"]
Nov 02 01:54:33.980130 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:33.980091    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-s7mp8"]
Nov 02 01:54:34.082426 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.082395    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:54:34.128845 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.128788    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:54:34.150548 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.150502    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:54:34.175109 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.175068    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f799283-3131-46a5-b010-628029210780-kube-api-access-5j88k\" (UniqueName: \"kubernetes.io/projected/1f799283-3131-46a5-b010-628029210780-kube-api-access-5j88k\") pod \"1f799283-3131-46a5-b010-628029210780\" (UID: \"1f799283-3131-46a5-b010-628029210780\") "
Nov 02 01:54:34.175361 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.175202    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-istio-data\") pod \"1f799283-3131-46a5-b010-628029210780\" (UID: \"1f799283-3131-46a5-b010-628029210780\") "
Nov 02 01:54:34.175361 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.175235    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-workload-certs\") pod \"1f799283-3131-46a5-b010-628029210780\" (UID: \"1f799283-3131-46a5-b010-628029210780\") "
Nov 02 01:54:34.175361 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.175315    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-workload-socket\") pod \"1f799283-3131-46a5-b010-628029210780\" (UID: \"1f799283-3131-46a5-b010-628029210780\") "
Nov 02 01:54:34.175574 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.175392    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f799283-3131-46a5-b010-628029210780-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f799283-3131-46a5-b010-628029210780-istio-podinfo\") pod \"1f799283-3131-46a5-b010-628029210780\" (UID: \"1f799283-3131-46a5-b010-628029210780\") "
Nov 02 01:54:34.175574 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.175422    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-istio-envoy\") pod \"1f799283-3131-46a5-b010-628029210780\" (UID: \"1f799283-3131-46a5-b010-628029210780\") "
Nov 02 01:54:34.175733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.175713    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f799283-3131-46a5-b010-628029210780-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f799283-3131-46a5-b010-628029210780-istiod-ca-cert\") pod \"1f799283-3131-46a5-b010-628029210780\" (UID: \"1f799283-3131-46a5-b010-628029210780\") "
Nov 02 01:54:34.175815 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.175799    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-credential-socket\") pod \"1f799283-3131-46a5-b010-628029210780\" (UID: \"1f799283-3131-46a5-b010-628029210780\") "
Nov 02 01:54:34.175876 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.175863    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f799283-3131-46a5-b010-628029210780-istio-token\" (UniqueName: \"kubernetes.io/projected/1f799283-3131-46a5-b010-628029210780-istio-token\") pod \"1f799283-3131-46a5-b010-628029210780\" (UID: \"1f799283-3131-46a5-b010-628029210780\") "
Nov 02 01:54:34.177371 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.177293    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-workload-socket" pod "1f799283-3131-46a5-b010-628029210780" (UID: "1f799283-3131-46a5-b010-628029210780"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:34.177567 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.177543    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-workload-certs" pod "1f799283-3131-46a5-b010-628029210780" (UID: "1f799283-3131-46a5-b010-628029210780"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:34.178712 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.178675    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-credential-socket" pod "1f799283-3131-46a5-b010-628029210780" (UID: "1f799283-3131-46a5-b010-628029210780"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:34.178981 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.178953    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f799283-3131-46a5-b010-628029210780-istiod-ca-cert" pod "1f799283-3131-46a5-b010-628029210780" (UID: "1f799283-3131-46a5-b010-628029210780"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:34.179373 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.179353    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-istio-data" pod "1f799283-3131-46a5-b010-628029210780" (UID: "1f799283-3131-46a5-b010-628029210780"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:34.180170 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.180145    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f799283-3131-46a5-b010-628029210780-istio-podinfo" pod "1f799283-3131-46a5-b010-628029210780" (UID: "1f799283-3131-46a5-b010-628029210780"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:54:34.180855 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.180832    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f799283-3131-46a5-b010-628029210780-kube-api-access-5j88k" pod "1f799283-3131-46a5-b010-628029210780" (UID: "1f799283-3131-46a5-b010-628029210780"). InnerVolumeSpecName "kube-api-access-5j88k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:34.182620 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.182598    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-istio-envoy" pod "1f799283-3131-46a5-b010-628029210780" (UID: "1f799283-3131-46a5-b010-628029210780"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:34.182620 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.182601    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f799283-3131-46a5-b010-628029210780-istio-token" pod "1f799283-3131-46a5-b010-628029210780" (UID: "1f799283-3131-46a5-b010-628029210780"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:34.213021 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.212940    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:54:34.277977 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.277946    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-workload-socket\") pod \"945be3db-df2f-4ef9-9609-a7890bee84ce\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") "
Nov 02 01:54:34.278160 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.277993    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-credential-socket\") pod \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") "
Nov 02 01:54:34.278160 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278017    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-workload-certs\") pod \"9984c750-63ba-4996-a4b1-44549030fe24\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") "
Nov 02 01:54:34.278160 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278066    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-credential-socket\") pod \"9984c750-63ba-4996-a4b1-44549030fe24\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") "
Nov 02 01:54:34.278160 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278099    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/945be3db-df2f-4ef9-9609-a7890bee84ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945be3db-df2f-4ef9-9609-a7890bee84ce-istiod-ca-cert\") pod \"945be3db-df2f-4ef9-9609-a7890bee84ce\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") "
Nov 02 01:54:34.278416 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278174    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-data\") pod \"945be3db-df2f-4ef9-9609-a7890bee84ce\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") "
Nov 02 01:54:34.278416 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278210    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-workload-certs\") pod \"945be3db-df2f-4ef9-9609-a7890bee84ce\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") "
Nov 02 01:54:34.278416 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278273    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-kube-api-access-22wr6\" (UniqueName: \"kubernetes.io/projected/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-kube-api-access-22wr6\") pod \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") "
Nov 02 01:54:34.278416 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278302    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-credential-socket" pod "1640ee78-dc59-4afb-a7f6-e130f60cdf4a" (UID: "1640ee78-dc59-4afb-a7f6-e130f60cdf4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:34.278416 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278269    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-workload-socket" pod "945be3db-df2f-4ef9-9609-a7890bee84ce" (UID: "945be3db-df2f-4ef9-9609-a7890bee84ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:34.278416 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278350    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-workload-socket\") pod \"9984c750-63ba-4996-a4b1-44549030fe24\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") "
Nov 02 01:54:34.278416 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278402    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-credential-socket" pod "9984c750-63ba-4996-a4b1-44549030fe24" (UID: "9984c750-63ba-4996-a4b1-44549030fe24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:34.278740 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278424    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9984c750-63ba-4996-a4b1-44549030fe24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9984c750-63ba-4996-a4b1-44549030fe24-istiod-ca-cert\") pod \"9984c750-63ba-4996-a4b1-44549030fe24\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") "
Nov 02 01:54:34.278740 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278478    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-data\") pod \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") "
Nov 02 01:54:34.278740 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278496    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-data" pod "945be3db-df2f-4ef9-9609-a7890bee84ce" (UID: "945be3db-df2f-4ef9-9609-a7890bee84ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:34.278740 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278520    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9984c750-63ba-4996-a4b1-44549030fe24-kube-api-access-vl2v9\" (UniqueName: \"kubernetes.io/projected/9984c750-63ba-4996-a4b1-44549030fe24-kube-api-access-vl2v9\") pod \"9984c750-63ba-4996-a4b1-44549030fe24\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") "
Nov 02 01:54:34.278740 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278553    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-envoy\") pod \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") "
Nov 02 01:54:34.278740 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278597    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9984c750-63ba-4996-a4b1-44549030fe24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9984c750-63ba-4996-a4b1-44549030fe24-istio-podinfo\") pod \"9984c750-63ba-4996-a4b1-44549030fe24\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") "
Nov 02 01:54:34.278740 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278652    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-token\" (UniqueName: \"kubernetes.io/projected/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-token\") pod \"945be3db-df2f-4ef9-9609-a7890bee84ce\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") "
Nov 02 01:54:34.278740 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278667    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-workload-socket" pod "9984c750-63ba-4996-a4b1-44549030fe24" (UID: "9984c750-63ba-4996-a4b1-44549030fe24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:34.278740 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278677    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-podinfo\") pod \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") "
Nov 02 01:54:34.278740 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278723    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-data" pod "1640ee78-dc59-4afb-a7f6-e130f60cdf4a" (UID: "1640ee78-dc59-4afb-a7f6-e130f60cdf4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:34.278740 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278725    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-workload-certs" pod "945be3db-df2f-4ef9-9609-a7890bee84ce" (UID: "945be3db-df2f-4ef9-9609-a7890bee84ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:34.278740 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278742    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-credential-socket\") pod \"945be3db-df2f-4ef9-9609-a7890bee84ce\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") "
Nov 02 01:54:34.279372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278773    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9984c750-63ba-4996-a4b1-44549030fe24-istio-token\" (UniqueName: \"kubernetes.io/projected/9984c750-63ba-4996-a4b1-44549030fe24-istio-token\") pod \"9984c750-63ba-4996-a4b1-44549030fe24\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") "
Nov 02 01:54:34.279372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278804    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/945be3db-df2f-4ef9-9609-a7890bee84ce-istiod-ca-cert" pod "945be3db-df2f-4ef9-9609-a7890bee84ce" (UID: "945be3db-df2f-4ef9-9609-a7890bee84ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:34.279372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278809    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istiod-ca-cert\") pod \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") "
Nov 02 01:54:34.279372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278822    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9984c750-63ba-4996-a4b1-44549030fe24-istiod-ca-cert" pod "9984c750-63ba-4996-a4b1-44549030fe24" (UID: "9984c750-63ba-4996-a4b1-44549030fe24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:34.279372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278850    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-workload-certs\") pod \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") "
Nov 02 01:54:34.279372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.278997    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-podinfo\") pod \"945be3db-df2f-4ef9-9609-a7890bee84ce\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") "
Nov 02 01:54:34.279372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.279111    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-envoy\") pod \"945be3db-df2f-4ef9-9609-a7890bee84ce\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") "
Nov 02 01:54:34.279372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.279145    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-workload-socket\") pod \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") "
Nov 02 01:54:34.279372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.279157    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-workload-certs" pod "9984c750-63ba-4996-a4b1-44549030fe24" (UID: "9984c750-63ba-4996-a4b1-44549030fe24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:34.279372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.279201    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istiod-ca-cert" pod "1640ee78-dc59-4afb-a7f6-e130f60cdf4a" (UID: "1640ee78-dc59-4afb-a7f6-e130f60cdf4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:34.279372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.279217    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945be3db-df2f-4ef9-9609-a7890bee84ce-kube-api-access-ncx8f\" (UniqueName: \"kubernetes.io/projected/945be3db-df2f-4ef9-9609-a7890bee84ce-kube-api-access-ncx8f\") pod \"945be3db-df2f-4ef9-9609-a7890bee84ce\" (UID: \"945be3db-df2f-4ef9-9609-a7890bee84ce\") "
Nov 02 01:54:34.279372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.279251    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-istio-data\") pod \"9984c750-63ba-4996-a4b1-44549030fe24\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") "
Nov 02 01:54:34.279372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.279306    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-istio-envoy\") pod \"9984c750-63ba-4996-a4b1-44549030fe24\" (UID: \"9984c750-63ba-4996-a4b1-44549030fe24\") "
Nov 02 01:54:34.279372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.279362    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-token\" (UniqueName: \"kubernetes.io/projected/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-token\") pod \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\" (UID: \"1640ee78-dc59-4afb-a7f6-e130f60cdf4a\") "
Nov 02 01:54:34.280291 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.279442    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-credential-socket" pod "945be3db-df2f-4ef9-9609-a7890bee84ce" (UID: "945be3db-df2f-4ef9-9609-a7890bee84ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:34.280291 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.279723    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-workload-certs" pod "1640ee78-dc59-4afb-a7f6-e130f60cdf4a" (UID: "1640ee78-dc59-4afb-a7f6-e130f60cdf4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:34.280291 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.280174    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-workload-socket" pod "1640ee78-dc59-4afb-a7f6-e130f60cdf4a" (UID: "1640ee78-dc59-4afb-a7f6-e130f60cdf4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:34.281171 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.280933    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-istio-data" pod "9984c750-63ba-4996-a4b1-44549030fe24" (UID: "9984c750-63ba-4996-a4b1-44549030fe24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:34.281171 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.281092    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.281171 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.281121    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f799283-3131-46a5-b010-628029210780-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.281171 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.281137    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.281171 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.281151    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.281171 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.281166    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.281581 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.281180    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.281581 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.281195    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945be3db-df2f-4ef9-9609-a7890bee84ce-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.281581 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.281212    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f799283-3131-46a5-b010-628029210780-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.281581 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.281227    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.281581 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.281241    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.281581 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.281256    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.281581 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.281271    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9984c750-63ba-4996-a4b1-44549030fe24-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.281581 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.281302    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.281581 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.281317    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j88k\" (UniqueName: \"kubernetes.io/projected/1f799283-3131-46a5-b010-628029210780-kube-api-access-5j88k\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.281581 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.281487    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-kube-api-access-22wr6" pod "1640ee78-dc59-4afb-a7f6-e130f60cdf4a" (UID: "1640ee78-dc59-4afb-a7f6-e130f60cdf4a"). InnerVolumeSpecName "kube-api-access-22wr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:34.282095 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.281670    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.282095 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.281693    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.282095 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.281708    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.282095 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.281734    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f799283-3131-46a5-b010-628029210780-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.282095 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.281749    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f799283-3131-46a5-b010-628029210780-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.282095 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.281763    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.282095 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.281777    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.282095 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.281789    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.282095 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.281802    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.282668 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.282634    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-podinfo" pod "945be3db-df2f-4ef9-9609-a7890bee84ce" (UID: "945be3db-df2f-4ef9-9609-a7890bee84ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:54:34.283062 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.283032    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-podinfo" pod "1640ee78-dc59-4afb-a7f6-e130f60cdf4a" (UID: "1640ee78-dc59-4afb-a7f6-e130f60cdf4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:54:34.283441 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.283417    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-token" pod "945be3db-df2f-4ef9-9609-a7890bee84ce" (UID: "945be3db-df2f-4ef9-9609-a7890bee84ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:34.283670 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.283631    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945be3db-df2f-4ef9-9609-a7890bee84ce-kube-api-access-ncx8f" pod "945be3db-df2f-4ef9-9609-a7890bee84ce" (UID: "945be3db-df2f-4ef9-9609-a7890bee84ce"). InnerVolumeSpecName "kube-api-access-ncx8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:34.283776 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.283722    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9984c750-63ba-4996-a4b1-44549030fe24-kube-api-access-vl2v9" pod "9984c750-63ba-4996-a4b1-44549030fe24" (UID: "9984c750-63ba-4996-a4b1-44549030fe24"). InnerVolumeSpecName "kube-api-access-vl2v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:34.283852 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.283800    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-envoy" pod "945be3db-df2f-4ef9-9609-a7890bee84ce" (UID: "945be3db-df2f-4ef9-9609-a7890bee84ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:34.283852 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.283799    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-envoy" pod "1640ee78-dc59-4afb-a7f6-e130f60cdf4a" (UID: "1640ee78-dc59-4afb-a7f6-e130f60cdf4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:34.283852 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.283810    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9984c750-63ba-4996-a4b1-44549030fe24-istio-token" pod "9984c750-63ba-4996-a4b1-44549030fe24" (UID: "9984c750-63ba-4996-a4b1-44549030fe24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:34.284433 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.284408    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-istio-envoy" pod "9984c750-63ba-4996-a4b1-44549030fe24" (UID: "9984c750-63ba-4996-a4b1-44549030fe24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:34.284546 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.284530    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-token" pod "1640ee78-dc59-4afb-a7f6-e130f60cdf4a" (UID: "1640ee78-dc59-4afb-a7f6-e130f60cdf4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:34.284792 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.284777    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9984c750-63ba-4996-a4b1-44549030fe24-istio-podinfo" pod "9984c750-63ba-4996-a4b1-44549030fe24" (UID: "9984c750-63ba-4996-a4b1-44549030fe24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:54:34.383556 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.383460    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22wr6\" (UniqueName: \"kubernetes.io/projected/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-kube-api-access-22wr6\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.383556 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.383495    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl2v9\" (UniqueName: \"kubernetes.io/projected/9984c750-63ba-4996-a4b1-44549030fe24-kube-api-access-vl2v9\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.383556 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.383505    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.383556 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.383516    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9984c750-63ba-4996-a4b1-44549030fe24-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.383556 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.383525    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.383556 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.383534    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.383556 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.383543    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9984c750-63ba-4996-a4b1-44549030fe24-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.383556 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.383552    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.383556 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.383562    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945be3db-df2f-4ef9-9609-a7890bee84ce-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.383958 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.383570    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncx8f\" (UniqueName: \"kubernetes.io/projected/945be3db-df2f-4ef9-9609-a7890bee84ce-kube-api-access-ncx8f\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.383958 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.383579    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.383958 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.383587    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9984c750-63ba-4996-a4b1-44549030fe24-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.383958 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.383595    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1640ee78-dc59-4afb-a7f6-e130f60cdf4a-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.592634 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.592600    5639 generic.go:356] "Generic (PLEG): container finished" podID="1f799283-3131-46a5-b010-628029210780" containerID="19b8c99f9f3e6a460f55fbc5880b434a658cea21501e8c09be3c18e9d42e1b2a" exitCode=0
Nov 02 01:54:34.593086 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.592652    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"
Nov 02 01:54:34.593086 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.592684    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz" event={"ID":"1f799283-3131-46a5-b010-628029210780","Type":"ContainerDied","Data":"19b8c99f9f3e6a460f55fbc5880b434a658cea21501e8c09be3c18e9d42e1b2a"}
Nov 02 01:54:34.593086 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.592752    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz" event={"ID":"1f799283-3131-46a5-b010-628029210780","Type":"ContainerDied","Data":"aeac582c7bb9ddb6a5835911f771b1c643c56fe99155bbd47dc40384e72735fc"}
Nov 02 01:54:34.593086 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.592786    5639 scope.go:122] "RemoveContainer" containerID="19b8c99f9f3e6a460f55fbc5880b434a658cea21501e8c09be3c18e9d42e1b2a"
Nov 02 01:54:34.595183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.595143    5639 generic.go:356] "Generic (PLEG): container finished" podID="9984c750-63ba-4996-a4b1-44549030fe24" containerID="f62dd4fc2e17f5b9b6c749612001a52f659c6a2f5f9fc324725bdab67d48a4e6" exitCode=0
Nov 02 01:54:34.595581 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.595207    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"
Nov 02 01:54:34.596063 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.595247    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5" event={"ID":"9984c750-63ba-4996-a4b1-44549030fe24","Type":"ContainerDied","Data":"f62dd4fc2e17f5b9b6c749612001a52f659c6a2f5f9fc324725bdab67d48a4e6"}
Nov 02 01:54:34.596273 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.596253    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5" event={"ID":"9984c750-63ba-4996-a4b1-44549030fe24","Type":"ContainerDied","Data":"e86c33e8f80d06c8250ce38ebf33c6a4b63c786bcdb48a637855a38191a57d36"}
Nov 02 01:54:34.607946 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.607916    5639 generic.go:356] "Generic (PLEG): container finished" podID="1640ee78-dc59-4afb-a7f6-e130f60cdf4a" containerID="9ac49c154511182455be13f90b691f0238fa39d2a62e516a02dd8e425f2e0a64" exitCode=0
Nov 02 01:54:34.608033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.607986    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"
Nov 02 01:54:34.608033 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.608000    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h" event={"ID":"1640ee78-dc59-4afb-a7f6-e130f60cdf4a","Type":"ContainerDied","Data":"9ac49c154511182455be13f90b691f0238fa39d2a62e516a02dd8e425f2e0a64"}
Nov 02 01:54:34.608151 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.608083    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h" event={"ID":"1640ee78-dc59-4afb-a7f6-e130f60cdf4a","Type":"ContainerDied","Data":"b8b5b5e5fb58f22f5f63b708f3448eb10bf673334181aceaa11ba13882bf72a0"}
Nov 02 01:54:34.610132 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.610103    5639 generic.go:356] "Generic (PLEG): container finished" podID="945be3db-df2f-4ef9-9609-a7890bee84ce" containerID="e07eb0ed26cff5e32d54db412e9bea74ff8f87d80a1aceed083e11651b35168e" exitCode=0
Nov 02 01:54:34.610369 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.610145    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp" event={"ID":"945be3db-df2f-4ef9-9609-a7890bee84ce","Type":"ContainerDied","Data":"e07eb0ed26cff5e32d54db412e9bea74ff8f87d80a1aceed083e11651b35168e"}
Nov 02 01:54:34.610369 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.610184    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp" event={"ID":"945be3db-df2f-4ef9-9609-a7890bee84ce","Type":"ContainerDied","Data":"8d0808082e697ffbf18b3eff4fbc3e2aa1ffc29100914c5a4e9208c14ed57d50"}
Nov 02 01:54:34.610369 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.610247    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"
Nov 02 01:54:34.618159 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.617918    5639 scope.go:122] "RemoveContainer" containerID="19b8c99f9f3e6a460f55fbc5880b434a658cea21501e8c09be3c18e9d42e1b2a"
Nov 02 01:54:34.618514 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:54:34.618421    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19b8c99f9f3e6a460f55fbc5880b434a658cea21501e8c09be3c18e9d42e1b2a\": container with ID starting with 19b8c99f9f3e6a460f55fbc5880b434a658cea21501e8c09be3c18e9d42e1b2a not found: ID does not exist" containerID="19b8c99f9f3e6a460f55fbc5880b434a658cea21501e8c09be3c18e9d42e1b2a"
Nov 02 01:54:34.618514 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.618466    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19b8c99f9f3e6a460f55fbc5880b434a658cea21501e8c09be3c18e9d42e1b2a"} err="failed to get container status \"19b8c99f9f3e6a460f55fbc5880b434a658cea21501e8c09be3c18e9d42e1b2a\": rpc error: code = NotFound desc = could not find container \"19b8c99f9f3e6a460f55fbc5880b434a658cea21501e8c09be3c18e9d42e1b2a\": container with ID starting with 19b8c99f9f3e6a460f55fbc5880b434a658cea21501e8c09be3c18e9d42e1b2a not found: ID does not exist"
Nov 02 01:54:34.618514 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.618496    5639 scope.go:122] "RemoveContainer" containerID="f62dd4fc2e17f5b9b6c749612001a52f659c6a2f5f9fc324725bdab67d48a4e6"
Nov 02 01:54:34.633816 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.633762    5639 scope.go:122] "RemoveContainer" containerID="f62dd4fc2e17f5b9b6c749612001a52f659c6a2f5f9fc324725bdab67d48a4e6"
Nov 02 01:54:34.634146 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:54:34.634125    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f62dd4fc2e17f5b9b6c749612001a52f659c6a2f5f9fc324725bdab67d48a4e6\": container with ID starting with f62dd4fc2e17f5b9b6c749612001a52f659c6a2f5f9fc324725bdab67d48a4e6 not found: ID does not exist" containerID="f62dd4fc2e17f5b9b6c749612001a52f659c6a2f5f9fc324725bdab67d48a4e6"
Nov 02 01:54:34.634213 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.634155    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f62dd4fc2e17f5b9b6c749612001a52f659c6a2f5f9fc324725bdab67d48a4e6"} err="failed to get container status \"f62dd4fc2e17f5b9b6c749612001a52f659c6a2f5f9fc324725bdab67d48a4e6\": rpc error: code = NotFound desc = could not find container \"f62dd4fc2e17f5b9b6c749612001a52f659c6a2f5f9fc324725bdab67d48a4e6\": container with ID starting with f62dd4fc2e17f5b9b6c749612001a52f659c6a2f5f9fc324725bdab67d48a4e6 not found: ID does not exist"
Nov 02 01:54:34.634213 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.634172    5639 scope.go:122] "RemoveContainer" containerID="9ac49c154511182455be13f90b691f0238fa39d2a62e516a02dd8e425f2e0a64"
Nov 02 01:54:34.645916 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.645892    5639 scope.go:122] "RemoveContainer" containerID="9ac49c154511182455be13f90b691f0238fa39d2a62e516a02dd8e425f2e0a64"
Nov 02 01:54:34.646353 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:54:34.646326    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ac49c154511182455be13f90b691f0238fa39d2a62e516a02dd8e425f2e0a64\": container with ID starting with 9ac49c154511182455be13f90b691f0238fa39d2a62e516a02dd8e425f2e0a64 not found: ID does not exist" containerID="9ac49c154511182455be13f90b691f0238fa39d2a62e516a02dd8e425f2e0a64"
Nov 02 01:54:34.646454 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.646364    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac49c154511182455be13f90b691f0238fa39d2a62e516a02dd8e425f2e0a64"} err="failed to get container status \"9ac49c154511182455be13f90b691f0238fa39d2a62e516a02dd8e425f2e0a64\": rpc error: code = NotFound desc = could not find container \"9ac49c154511182455be13f90b691f0238fa39d2a62e516a02dd8e425f2e0a64\": container with ID starting with 9ac49c154511182455be13f90b691f0238fa39d2a62e516a02dd8e425f2e0a64 not found: ID does not exist"
Nov 02 01:54:34.646454 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.646388    5639 scope.go:122] "RemoveContainer" containerID="e07eb0ed26cff5e32d54db412e9bea74ff8f87d80a1aceed083e11651b35168e"
Nov 02 01:54:34.659619 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.659592    5639 scope.go:122] "RemoveContainer" containerID="e07eb0ed26cff5e32d54db412e9bea74ff8f87d80a1aceed083e11651b35168e"
Nov 02 01:54:34.660007 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:54:34.659983    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e07eb0ed26cff5e32d54db412e9bea74ff8f87d80a1aceed083e11651b35168e\": container with ID starting with e07eb0ed26cff5e32d54db412e9bea74ff8f87d80a1aceed083e11651b35168e not found: ID does not exist" containerID="e07eb0ed26cff5e32d54db412e9bea74ff8f87d80a1aceed083e11651b35168e"
Nov 02 01:54:34.660079 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.660016    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07eb0ed26cff5e32d54db412e9bea74ff8f87d80a1aceed083e11651b35168e"} err="failed to get container status \"e07eb0ed26cff5e32d54db412e9bea74ff8f87d80a1aceed083e11651b35168e\": rpc error: code = NotFound desc = could not find container \"e07eb0ed26cff5e32d54db412e9bea74ff8f87d80a1aceed083e11651b35168e\": container with ID starting with e07eb0ed26cff5e32d54db412e9bea74ff8f87d80a1aceed083e11651b35168e not found: ID does not exist"
Nov 02 01:54:34.783306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.776460    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"]
Nov 02 01:54:34.815304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.812266    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-x2bbz"]
Nov 02 01:54:34.897153 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.896709    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"]
Nov 02 01:54:34.905695 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.904482    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-dgjgp"]
Nov 02 01:54:34.991900 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.991839    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"]
Nov 02 01:54:34.996770 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:34.996741    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-rg68h"]
Nov 02 01:54:35.110763 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:35.110731    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"]
Nov 02 01:54:35.117856 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:35.117828    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-4lgg5"]
Nov 02 01:54:35.222005 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:35.221930    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1640ee78-dc59-4afb-a7f6-e130f60cdf4a" path="/var/lib/kubelet/pods/1640ee78-dc59-4afb-a7f6-e130f60cdf4a/volumes"
Nov 02 01:54:35.222492 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:35.222476    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f799283-3131-46a5-b010-628029210780" path="/var/lib/kubelet/pods/1f799283-3131-46a5-b010-628029210780/volumes"
Nov 02 01:54:35.222933 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:35.222919    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3da5e6df-b15c-4209-9cec-624599f6713d" path="/var/lib/kubelet/pods/3da5e6df-b15c-4209-9cec-624599f6713d/volumes"
Nov 02 01:54:35.223708 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:35.223691    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="945be3db-df2f-4ef9-9609-a7890bee84ce" path="/var/lib/kubelet/pods/945be3db-df2f-4ef9-9609-a7890bee84ce/volumes"
Nov 02 01:54:35.224326 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:35.224303    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9984c750-63ba-4996-a4b1-44549030fe24" path="/var/lib/kubelet/pods/9984c750-63ba-4996-a4b1-44549030fe24/volumes"
Nov 02 01:54:35.224954 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:35.224938    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ab0dca-9dde-45ae-8647-61a346c461f2" path="/var/lib/kubelet/pods/e5ab0dca-9dde-45ae-8647-61a346c461f2/volumes"
Nov 02 01:54:35.225703 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:35.225686    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb887678-0155-4d81-be15-861a7e1f8ab8" path="/var/lib/kubelet/pods/fb887678-0155-4d81-be15-861a7e1f8ab8/volumes"
Nov 02 01:54:37.013325 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:37.013290    5639 scope.go:122] "RemoveContainer" containerID="9e1c6b166e4853892eea71044805c738ad7461382af41f642c9cf73fcef4bafd"
Nov 02 01:54:37.024606 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:37.024584    5639 scope.go:122] "RemoveContainer" containerID="f7954afe2c6f0f03d1eaed6cd489ea08e0db4b89a55f448d18334a45b5aae657"
Nov 02 01:54:37.035558 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:37.035528    5639 scope.go:122] "RemoveContainer" containerID="90dbf4cfdebc679f3ebcb9c7e0d7b8691ced8d6c84d963242af85dc66753b5bc"
Nov 02 01:54:37.046700 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:37.046672    5639 scope.go:122] "RemoveContainer" containerID="eb16ce431579c03ae2cbda2f4fa7f840074007c081324cd684ac24c103f7a9a5"
Nov 02 01:54:37.057786 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:37.057763    5639 scope.go:122] "RemoveContainer" containerID="ccbd60394c6c0b2e3c0ff2b50dd2884b64af485ad7384701493ff8defa26e5ed"
Nov 02 01:54:37.067978 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:37.067957    5639 scope.go:122] "RemoveContainer" containerID="d7835ae8e56c148a0b1e7be0ed2e4238bd9d16bc118d27a618531bdfd1946bcd"
Nov 02 01:54:37.078492 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:37.078468    5639 scope.go:122] "RemoveContainer" containerID="0dc16592262cc54e5da0c9c56113317f431f5f6633e6478f68237bd3e71d0f37"
Nov 02 01:54:37.088652 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:37.088631    5639 scope.go:122] "RemoveContainer" containerID="a0cf4436ee4cca56be1c87491c13df298da4d36a20bcd866236799a7d7ba5d72"
Nov 02 01:54:37.098875 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:37.098855    5639 scope.go:122] "RemoveContainer" containerID="05b0aa7e692003c7df5be175feb0d8ab24d07b14c2c83b38bac732565786d45e"
Nov 02 01:54:37.109265 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:37.109243    5639 scope.go:122] "RemoveContainer" containerID="cbc6f984ed172f9e9fe6d0e992bde1a1629874dbefa7ad9553e94e3d5d30d816"
Nov 02 01:54:37.119670 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:37.119646    5639 scope.go:122] "RemoveContainer" containerID="bf360c2b5bb56c9787c73e732b381dcf7d444afd0840f5a0c5c1c87f1d008535"
Nov 02 01:54:37.129869 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:37.129850    5639 scope.go:122] "RemoveContainer" containerID="da588cf5782dd1ab6937881732c33253109b92e3d5c104f0a92db225266365e9"
Nov 02 01:54:37.141209 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:37.141184    5639 scope.go:122] "RemoveContainer" containerID="8049a0bd26e1297fcf415172077af6a4ccf093155e08bf64d87dcf7b2b46ab87"
Nov 02 01:54:37.153036 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:37.153009    5639 scope.go:122] "RemoveContainer" containerID="4ba4b9968b53bdc424e033da577f8f3bf7f1050b126a8c61acb3b7c795db726c"
Nov 02 01:54:37.163540 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:37.163520    5639 scope.go:122] "RemoveContainer" containerID="141964ec7fb312d76de05c700dca19346c8da659ca318fe11398da1d30b9b206"
Nov 02 01:54:41.058640 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:41.058590    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ffb490f-8tplc"]
Nov 02 01:54:41.126692 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:41.126650    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ffb490f-8tplc"]
Nov 02 01:54:41.232055 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:41.232016    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="093e8792-ba76-40e9-aad5-90b314440ca8" path="/var/lib/kubelet/pods/093e8792-ba76-40e9-aad5-90b314440ca8/volumes"
Nov 02 01:54:43.490530 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:43.490475    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:54:43.760490 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:43.760382    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:44.010332 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:44.010254    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:44.011056 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:44.010979    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:44.011582 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:44.011557    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:54:44.050061 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:44.050001    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 02 01:54:44.052516 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:44.052425    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:44.159578 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:44.159264    5639 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:44.176777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:44.176733    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:44.225448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:44.225403    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:44.492309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:44.491397    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:44.492309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:44.492010    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:54:44.582602 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:44.582568    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 01:54:44.761359 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:44.761237    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:44.763383 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:44.761711    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:54:45.013670 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:45.013196    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:45.013935 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:45.013894    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:45.025428 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:45.014272    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Nov 02 01:54:45.025428 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:45.014385    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:54:45.025428 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:45.014920    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:54:45.025428 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:45.014959    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:54:45.037700 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:45.037673    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 01:54:45.055646 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:45.055608    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:45.056335 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:45.056312    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:54:45.137852 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:45.137816    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 01:54:45.149619 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:45.143992    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 01:54:45.149619 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:45.145721    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 01:54:45.149619 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:45.146472    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 01:54:50.353463 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:50.353415    5639 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:54:50.353968 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:50.353530    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8" podUID="ceaab0ba-9b59-4a1a-839d-bbd4322f9629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:51.048870 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:51.048784    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2804cba7-8hgcq"]
Nov 02 01:54:51.098063 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:51.098029    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2804cba7-8hgcq"]
Nov 02 01:54:51.220696 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:51.220656    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a" path="/var/lib/kubelet/pods/bbde1c6d-b9cf-4366-b5cf-94ce70ac0a2a/volumes"
Nov 02 01:54:51.694028 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:51.682418    5639 generic.go:356] "Generic (PLEG): container finished" podID="ceaab0ba-9b59-4a1a-839d-bbd4322f9629" containerID="aff79c7033179a98e70ec2051073a3ee7ad68c5c722173160fa4124cb1f1135b" exitCode=0
Nov 02 01:54:51.694028 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:51.682548    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8" event={"ID":"ceaab0ba-9b59-4a1a-839d-bbd4322f9629","Type":"ContainerDied","Data":"aff79c7033179a98e70ec2051073a3ee7ad68c5c722173160fa4124cb1f1135b"}
Nov 02 01:54:51.694028 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:51.683071    5639 scope.go:122] "RemoveContainer" containerID="aff79c7033179a98e70ec2051073a3ee7ad68c5c722173160fa4124cb1f1135b"
Nov 02 01:54:51.694028 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:51.693970    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:54:51.695113 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:51.694072    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:54:51.695113 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:51.694146    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.099859    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg"]
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.100890    5639 cpu_manager.go:414] "containerMap: removing container" podUID="e5ab0dca-9dde-45ae-8647-61a346c461f2" containerName="istio-proxy"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.100911    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ab0dca-9dde-45ae-8647-61a346c461f2" containerName="istio-proxy"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.100929    5639 cpu_manager.go:414] "containerMap: removing container" podUID="d8ba9d0e-66ce-4927-aed2-e84b3280e116" containerName="istio-proxy"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.100937    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8ba9d0e-66ce-4927-aed2-e84b3280e116" containerName="istio-proxy"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.100951    5639 cpu_manager.go:414] "containerMap: removing container" podUID="3da5e6df-b15c-4209-9cec-624599f6713d" containerName="istio-proxy"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.100958    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3da5e6df-b15c-4209-9cec-624599f6713d" containerName="istio-proxy"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.100987    5639 cpu_manager.go:414] "containerMap: removing container" podUID="945be3db-df2f-4ef9-9609-a7890bee84ce" containerName="istio-proxy"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.100995    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945be3db-df2f-4ef9-9609-a7890bee84ce" containerName="istio-proxy"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.101020    5639 cpu_manager.go:414] "containerMap: removing container" podUID="fb887678-0155-4d81-be15-861a7e1f8ab8" containerName="istio-proxy"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.101028    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb887678-0155-4d81-be15-861a7e1f8ab8" containerName="istio-proxy"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.101042    5639 cpu_manager.go:414] "containerMap: removing container" podUID="9984c750-63ba-4996-a4b1-44549030fe24" containerName="istio-proxy"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.101049    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9984c750-63ba-4996-a4b1-44549030fe24" containerName="istio-proxy"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.101096    5639 cpu_manager.go:414] "containerMap: removing container" podUID="d8ba9d0e-66ce-4927-aed2-e84b3280e116" containerName="istio-proxy"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.101104    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8ba9d0e-66ce-4927-aed2-e84b3280e116" containerName="istio-proxy"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.101116    5639 cpu_manager.go:414] "containerMap: removing container" podUID="1f799283-3131-46a5-b010-628029210780" containerName="istio-proxy"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.101124    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f799283-3131-46a5-b010-628029210780" containerName="istio-proxy"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.101136    5639 cpu_manager.go:414] "containerMap: removing container" podUID="1640ee78-dc59-4afb-a7f6-e130f60cdf4a" containerName="istio-proxy"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.101142    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1640ee78-dc59-4afb-a7f6-e130f60cdf4a" containerName="istio-proxy"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.101152    5639 cpu_manager.go:414] "containerMap: removing container" podUID="3b04800a-455b-497e-b3a9-7ee75f8e62f6" containerName="revoke-keys"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.101159    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b04800a-455b-497e-b3a9-7ee75f8e62f6" containerName="revoke-keys"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.101324    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="1640ee78-dc59-4afb-a7f6-e130f60cdf4a" containerName="istio-proxy"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.101338    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b04800a-455b-497e-b3a9-7ee75f8e62f6" containerName="revoke-keys"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.101349    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ab0dca-9dde-45ae-8647-61a346c461f2" containerName="istio-proxy"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.101357    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8ba9d0e-66ce-4927-aed2-e84b3280e116" containerName="istio-proxy"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.101366    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="3da5e6df-b15c-4209-9cec-624599f6713d" containerName="istio-proxy"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.101379    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="945be3db-df2f-4ef9-9609-a7890bee84ce" containerName="istio-proxy"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.101388    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb887678-0155-4d81-be15-861a7e1f8ab8" containerName="istio-proxy"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.101396    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="9984c750-63ba-4996-a4b1-44549030fe24" containerName="istio-proxy"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.101413    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8ba9d0e-66ce-4927-aed2-e84b3280e116" containerName="istio-proxy"
Nov 02 01:54:52.116448 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.101422    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f799283-3131-46a5-b010-628029210780" containerName="istio-proxy"
Nov 02 01:54:52.182824 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.181493    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg"
Nov 02 01:54:52.276337 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.257084    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-07ac70d0\""
Nov 02 01:54:52.276655 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.276624    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-07ac70d0-dockercfg-q6h5f\""
Nov 02 01:54:52.287234 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.287198    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg"]
Nov 02 01:54:52.302524 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.296513    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw"]
Nov 02 01:54:52.304319 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.303938    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6424735d-18ec-462d-bb38-104055e1e001-plugins-config-volume\") pod \"payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg\" (UID: \"6424735d-18ec-462d-bb38-104055e1e001\") " pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg"
Nov 02 01:54:52.304319 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.304115    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgrkz\" (UniqueName: \"kubernetes.io/projected/6424735d-18ec-462d-bb38-104055e1e001-kube-api-access-dgrkz\") pod \"payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg\" (UID: \"6424735d-18ec-462d-bb38-104055e1e001\") " pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg"
Nov 02 01:54:52.317162 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.315544    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw"]
Nov 02 01:54:52.317162 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.315668    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw"
Nov 02 01:54:52.405642 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.405604    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6424735d-18ec-462d-bb38-104055e1e001-plugins-config-volume\") pod \"payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg\" (UID: \"6424735d-18ec-462d-bb38-104055e1e001\") " pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg"
Nov 02 01:54:52.405807 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.405763    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7157f0f1-b793-41ee-ab6a-cdf56fd7d72c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw\" (UID: \"7157f0f1-b793-41ee-ab6a-cdf56fd7d72c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw"
Nov 02 01:54:52.405857 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.405817    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgrkz\" (UniqueName: \"kubernetes.io/projected/6424735d-18ec-462d-bb38-104055e1e001-kube-api-access-dgrkz\") pod \"payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg\" (UID: \"6424735d-18ec-462d-bb38-104055e1e001\") " pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg"
Nov 02 01:54:52.405980 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.405955    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk6hp\" (UniqueName: \"kubernetes.io/projected/7157f0f1-b793-41ee-ab6a-cdf56fd7d72c-kube-api-access-nk6hp\") pod \"payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw\" (UID: \"7157f0f1-b793-41ee-ab6a-cdf56fd7d72c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw"
Nov 02 01:54:52.406443 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.406426    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6424735d-18ec-462d-bb38-104055e1e001-plugins-config-volume\") pod \"payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg\" (UID: \"6424735d-18ec-462d-bb38-104055e1e001\") " pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg"
Nov 02 01:54:52.467338 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.467298    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgrkz\" (UniqueName: \"kubernetes.io/projected/6424735d-18ec-462d-bb38-104055e1e001-kube-api-access-dgrkz\") pod \"payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg\" (UID: \"6424735d-18ec-462d-bb38-104055e1e001\") " pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg"
Nov 02 01:54:52.507807 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.507712    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7157f0f1-b793-41ee-ab6a-cdf56fd7d72c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw\" (UID: \"7157f0f1-b793-41ee-ab6a-cdf56fd7d72c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw"
Nov 02 01:54:52.507807 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.507797    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk6hp\" (UniqueName: \"kubernetes.io/projected/7157f0f1-b793-41ee-ab6a-cdf56fd7d72c-kube-api-access-nk6hp\") pod \"payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw\" (UID: \"7157f0f1-b793-41ee-ab6a-cdf56fd7d72c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw"
Nov 02 01:54:52.508573 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.508549    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7157f0f1-b793-41ee-ab6a-cdf56fd7d72c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw\" (UID: \"7157f0f1-b793-41ee-ab6a-cdf56fd7d72c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw"
Nov 02 01:54:52.568922 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.568886    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk6hp\" (UniqueName: \"kubernetes.io/projected/7157f0f1-b793-41ee-ab6a-cdf56fd7d72c-kube-api-access-nk6hp\") pod \"payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw\" (UID: \"7157f0f1-b793-41ee-ab6a-cdf56fd7d72c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw"
Nov 02 01:54:52.577750 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.577722    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg"
Nov 02 01:54:52.644490 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.644453    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw"
Nov 02 01:54:52.707382 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.707330    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8" event={"ID":"ceaab0ba-9b59-4a1a-839d-bbd4322f9629","Type":"ContainerStarted","Data":"baa3b1d8fb5a9a269982b4667258075b61ed6b57be50bb68153cf3c7f402f5a7"}
Nov 02 01:54:52.990081 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.990038    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg"]
Nov 02 01:54:52.991916 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:54:52.991876    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6424735d_18ec_462d_bb38_104055e1e001.slice/crio-fd65ea0ad42f8aab07de612e5d07f2aa4adfbb7316c33d17ede8190a7f8cf817 WatchSource:0}: Error finding container fd65ea0ad42f8aab07de612e5d07f2aa4adfbb7316c33d17ede8190a7f8cf817: Status 404 returned error can't find the container with id fd65ea0ad42f8aab07de612e5d07f2aa4adfbb7316c33d17ede8190a7f8cf817
Nov 02 01:54:52.992742 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:52.992706    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:54:53.042881 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:54:53.042309    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7157f0f1_b793_41ee_ab6a_cdf56fd7d72c.slice/crio-0b3e3359f9225ec21766ea44a22ca05cbefae443b54785863baf583e695f7c39 WatchSource:0}: Error finding container 0b3e3359f9225ec21766ea44a22ca05cbefae443b54785863baf583e695f7c39: Status 404 returned error can't find the container with id 0b3e3359f9225ec21766ea44a22ca05cbefae443b54785863baf583e695f7c39
Nov 02 01:54:53.043702 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:53.043675    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw"]
Nov 02 01:54:53.716701 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:53.716660    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw" event={"ID":"7157f0f1-b793-41ee-ab6a-cdf56fd7d72c","Type":"ContainerStarted","Data":"4bdd010ad62c5fad0216c6d3eb9c3f0b15eddbdbee8dcbf8424ebc23bc73b1c6"}
Nov 02 01:54:53.717186 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:53.716730    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw" event={"ID":"7157f0f1-b793-41ee-ab6a-cdf56fd7d72c","Type":"ContainerStarted","Data":"0b3e3359f9225ec21766ea44a22ca05cbefae443b54785863baf583e695f7c39"}
Nov 02 01:54:53.717186 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:53.716844    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw"
Nov 02 01:54:53.718575 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:53.718551    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg" event={"ID":"6424735d-18ec-462d-bb38-104055e1e001","Type":"ContainerStarted","Data":"66e07aaf5f8d155fc75336b313986f5ea311d5278e422134f7811c54de25e445"}
Nov 02 01:54:53.718704 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:53.718594    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg" event={"ID":"6424735d-18ec-462d-bb38-104055e1e001","Type":"ContainerStarted","Data":"fd65ea0ad42f8aab07de612e5d07f2aa4adfbb7316c33d17ede8190a7f8cf817"}
Nov 02 01:54:53.718771 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:53.718756    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg"
Nov 02 01:54:53.849187 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:53.849132    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw" podStartSLOduration=1.849118119 podStartE2EDuration="1.849118119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:53.847671557 +0000 UTC m=+1758.751353673" watchObservedRunningTime="2026-11-02 01:54:53.849118119 +0000 UTC m=+1758.752800232"
Nov 02 01:54:54.019351 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:54.018780    5639 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 01:54:54.019351 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:54.018846    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8" podUID="ceaab0ba-9b59-4a1a-839d-bbd4322f9629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 01:54:54.028302 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:54.025035    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg" podStartSLOduration=3.025011952 podStartE2EDuration="3.025011952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:53.963096094 +0000 UTC m=+1758.866778208" watchObservedRunningTime="2026-11-02 01:54:54.025011952 +0000 UTC m=+1758.928694070"
Nov 02 01:54:54.993718 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:54.993679    5639 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 01:54:54.994128 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:54.993790    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8" podUID="ceaab0ba-9b59-4a1a-839d-bbd4322f9629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 01:54:55.992668 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:55.992631    5639 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 01:54:55.992856 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:55.992689    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8" podUID="ceaab0ba-9b59-4a1a-839d-bbd4322f9629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 01:54:56.993061 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:56.993014    5639 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 01:54:56.993494 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:56.993092    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8" podUID="ceaab0ba-9b59-4a1a-839d-bbd4322f9629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 01:54:57.422746 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:54:57.422705    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:54:57.994051 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:57.993597    5639 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 01:54:57.994051 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:57.993658    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8" podUID="ceaab0ba-9b59-4a1a-839d-bbd4322f9629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 01:54:58.021095 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:58.021060    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"]
Nov 02 01:54:58.021274 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:58.021254    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4" podUID="2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb" containerName="istio-proxy" containerID="cri-o://8f2444dbcb8a20f1a1811ccec632e5faf22156fd547d64079fd789e259aeda5c" gracePeriod=30
Nov 02 01:54:58.998146 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:58.998112    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:54:58.998636 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:58.998617    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:54:58.999466 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:54:58.999446    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 01:55:01.904302 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:01.896209    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8"]
Nov 02 01:55:01.959595 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:01.959561    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8"
Nov 02 01:55:02.026013 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:02.025882    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx57b\" (UniqueName: \"kubernetes.io/projected/f35a24df-cd1c-4768-9596-3ef625d19864-kube-api-access-zx57b\") pod \"payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8\" (UID: \"f35a24df-cd1c-4768-9596-3ef625d19864\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8"
Nov 02 01:55:02.026013 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:02.025974    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f35a24df-cd1c-4768-9596-3ef625d19864-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8\" (UID: \"f35a24df-cd1c-4768-9596-3ef625d19864\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8"
Nov 02 01:55:02.097304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:02.094748    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-257466\""
Nov 02 01:55:02.097304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:02.095167    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-257466-dockercfg-zhn8p\""
Nov 02 01:55:02.114485 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:02.112375    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8"]
Nov 02 01:55:02.114485 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:02.112463    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg"]
Nov 02 01:55:02.119724 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:02.119147    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg"
Nov 02 01:55:02.122909 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:02.122879    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg"]
Nov 02 01:55:02.309316 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:02.304332    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qsqh\" (UniqueName: \"kubernetes.io/projected/d8e14587-d00a-43a2-8eba-b907a34a3d91-kube-api-access-6qsqh\") pod \"payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg\" (UID: \"d8e14587-d00a-43a2-8eba-b907a34a3d91\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg"
Nov 02 01:55:02.309316 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:02.304658    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx57b\" (UniqueName: \"kubernetes.io/projected/f35a24df-cd1c-4768-9596-3ef625d19864-kube-api-access-zx57b\") pod \"payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8\" (UID: \"f35a24df-cd1c-4768-9596-3ef625d19864\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8"
Nov 02 01:55:02.309316 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:02.304731    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f35a24df-cd1c-4768-9596-3ef625d19864-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8\" (UID: \"f35a24df-cd1c-4768-9596-3ef625d19864\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8"
Nov 02 01:55:02.309316 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:02.304767    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e14587-d00a-43a2-8eba-b907a34a3d91-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg\" (UID: \"d8e14587-d00a-43a2-8eba-b907a34a3d91\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg"
Nov 02 01:55:02.309316 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:02.306941    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f35a24df-cd1c-4768-9596-3ef625d19864-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8\" (UID: \"f35a24df-cd1c-4768-9596-3ef625d19864\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8"
Nov 02 01:55:02.368340 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:02.368298    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx57b\" (UniqueName: \"kubernetes.io/projected/f35a24df-cd1c-4768-9596-3ef625d19864-kube-api-access-zx57b\") pod \"payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8\" (UID: \"f35a24df-cd1c-4768-9596-3ef625d19864\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8"
Nov 02 01:55:02.413190 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:02.413151    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e14587-d00a-43a2-8eba-b907a34a3d91-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg\" (UID: \"d8e14587-d00a-43a2-8eba-b907a34a3d91\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg"
Nov 02 01:55:02.413403 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:02.413358    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qsqh\" (UniqueName: \"kubernetes.io/projected/d8e14587-d00a-43a2-8eba-b907a34a3d91-kube-api-access-6qsqh\") pod \"payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg\" (UID: \"d8e14587-d00a-43a2-8eba-b907a34a3d91\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg"
Nov 02 01:55:02.413919 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:02.413900    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e14587-d00a-43a2-8eba-b907a34a3d91-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg\" (UID: \"d8e14587-d00a-43a2-8eba-b907a34a3d91\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg"
Nov 02 01:55:02.442017 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:02.441982    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8"
Nov 02 01:55:02.474304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:02.474254    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qsqh\" (UniqueName: \"kubernetes.io/projected/d8e14587-d00a-43a2-8eba-b907a34a3d91-kube-api-access-6qsqh\") pod \"payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg\" (UID: \"d8e14587-d00a-43a2-8eba-b907a34a3d91\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg"
Nov 02 01:55:02.640874 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:02.640842    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg"
Nov 02 01:55:02.827548 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:02.827516    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8"]
Nov 02 01:55:02.828797 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:55:02.828767    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35a24df_cd1c_4768_9596_3ef625d19864.slice/crio-af1400295c6ca94ed96f0a19221c7d61a8a65ff97971011775f40ed325f65129 WatchSource:0}: Error finding container af1400295c6ca94ed96f0a19221c7d61a8a65ff97971011775f40ed325f65129: Status 404 returned error can't find the container with id af1400295c6ca94ed96f0a19221c7d61a8a65ff97971011775f40ed325f65129
Nov 02 01:55:02.919180 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:02.919146    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8" event={"ID":"f35a24df-cd1c-4768-9596-3ef625d19864","Type":"ContainerStarted","Data":"af1400295c6ca94ed96f0a19221c7d61a8a65ff97971011775f40ed325f65129"}
Nov 02 01:55:02.919678 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:02.919235    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8"
Nov 02 01:55:03.021623 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:55:03.021581    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e14587_d00a_43a2_8eba_b907a34a3d91.slice/crio-42b9c95fa87ae16d4b6c00ae190ded3bc6e7ea896178c86a084c2f6bb5e2d689 WatchSource:0}: Error finding container 42b9c95fa87ae16d4b6c00ae190ded3bc6e7ea896178c86a084c2f6bb5e2d689: Status 404 returned error can't find the container with id 42b9c95fa87ae16d4b6c00ae190ded3bc6e7ea896178c86a084c2f6bb5e2d689
Nov 02 01:55:03.021790 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.021772    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg"]
Nov 02 01:55:03.043271 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.041770    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8" podStartSLOduration=2.041751509 podStartE2EDuration="2.041751509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:03.035873354 +0000 UTC m=+1767.939555466" watchObservedRunningTime="2026-11-02 01:55:03.041751509 +0000 UTC m=+1767.945433621"
Nov 02 01:55:03.610693 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.610663    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:55:03.641816 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.641782    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:55:03.699796 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.699706    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istiod-ca-cert\") pod \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") "
Nov 02 01:55:03.699796 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.699755    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-credential-socket\") pod \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") "
Nov 02 01:55:03.699796 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.699782    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-token\" (UniqueName: \"kubernetes.io/projected/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-token\") pod \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") "
Nov 02 01:55:03.700095 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.699872    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-workload-certs\") pod \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") "
Nov 02 01:55:03.700095 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.699938    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-data\") pod \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") "
Nov 02 01:55:03.700095 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.699979    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-kube-api-access-5jhcs\" (UniqueName: \"kubernetes.io/projected/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-kube-api-access-5jhcs\") pod \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") "
Nov 02 01:55:03.700095 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.700009    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-workload-socket\") pod \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") "
Nov 02 01:55:03.700095 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.700034    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-envoy\") pod \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") "
Nov 02 01:55:03.700095 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.700039    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-credential-socket" pod "2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb" (UID: "2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:03.700095 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.700083    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-podinfo\") pod \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\" (UID: \"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb\") "
Nov 02 01:55:03.700585 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.700137    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-workload-certs" pod "2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb" (UID: "2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:03.700585 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.700153    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istiod-ca-cert" pod "2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb" (UID: "2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:55:03.700682 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.700601    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-data" pod "2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb" (UID: "2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:03.701038 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.701019    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:03.701139 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.701043    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:03.701139 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.701056    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:03.701139 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.701068    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:03.701716 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.701403    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-workload-socket" pod "2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb" (UID: "2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:03.702845 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.702810    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-podinfo" pod "2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb" (UID: "2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:55:03.702959 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.702841    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-token" pod "2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb" (UID: "2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:55:03.703024 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.702997    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-kube-api-access-5jhcs" pod "2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb" (UID: "2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb"). InnerVolumeSpecName "kube-api-access-5jhcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:55:03.703101 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.703083    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-envoy" pod "2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb" (UID: "2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:03.802717 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.802669    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:03.802717 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.802705    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jhcs\" (UniqueName: \"kubernetes.io/projected/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-kube-api-access-5jhcs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:03.802717 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.802717    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:03.802717 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.802726    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:03.802999 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.802734    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:03.928431 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.928385    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8" event={"ID":"f35a24df-cd1c-4768-9596-3ef625d19864","Type":"ContainerStarted","Data":"c096363a247a4161af13da34b3a653a74e8bd26e51df3a6cfd8e92245acc0a80"}
Nov 02 01:55:03.930160 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.930131    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg" event={"ID":"d8e14587-d00a-43a2-8eba-b907a34a3d91","Type":"ContainerStarted","Data":"3c664d046bbc02d4eba48cff4905bff72dec56f707646ccb1a4e928bc6afd8af"}
Nov 02 01:55:03.930330 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.930180    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg" event={"ID":"d8e14587-d00a-43a2-8eba-b907a34a3d91","Type":"ContainerStarted","Data":"42b9c95fa87ae16d4b6c00ae190ded3bc6e7ea896178c86a084c2f6bb5e2d689"}
Nov 02 01:55:03.930330 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.930209    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg"
Nov 02 01:55:03.931727 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.931702    5639 generic.go:356] "Generic (PLEG): container finished" podID="2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb" containerID="8f2444dbcb8a20f1a1811ccec632e5faf22156fd547d64079fd789e259aeda5c" exitCode=0
Nov 02 01:55:03.931847 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.931739    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"
Nov 02 01:55:03.931847 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.931784    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4" event={"ID":"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb","Type":"ContainerDied","Data":"8f2444dbcb8a20f1a1811ccec632e5faf22156fd547d64079fd789e259aeda5c"}
Nov 02 01:55:03.931847 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.931835    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4" event={"ID":"2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb","Type":"ContainerDied","Data":"9567639b1f97913f7991a73d9becd5e644a511a15da705182cf7c80c0d7fad94"}
Nov 02 01:55:03.932039 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.931864    5639 scope.go:122] "RemoveContainer" containerID="8f2444dbcb8a20f1a1811ccec632e5faf22156fd547d64079fd789e259aeda5c"
Nov 02 01:55:03.943975 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.943943    5639 scope.go:122] "RemoveContainer" containerID="8f2444dbcb8a20f1a1811ccec632e5faf22156fd547d64079fd789e259aeda5c"
Nov 02 01:55:03.944265 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:55:03.944248    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f2444dbcb8a20f1a1811ccec632e5faf22156fd547d64079fd789e259aeda5c\": container with ID starting with 8f2444dbcb8a20f1a1811ccec632e5faf22156fd547d64079fd789e259aeda5c not found: ID does not exist" containerID="8f2444dbcb8a20f1a1811ccec632e5faf22156fd547d64079fd789e259aeda5c"
Nov 02 01:55:03.944362 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:03.944274    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f2444dbcb8a20f1a1811ccec632e5faf22156fd547d64079fd789e259aeda5c"} err="failed to get container status \"8f2444dbcb8a20f1a1811ccec632e5faf22156fd547d64079fd789e259aeda5c\": rpc error: code = NotFound desc = could not find container \"8f2444dbcb8a20f1a1811ccec632e5faf22156fd547d64079fd789e259aeda5c\": container with ID starting with 8f2444dbcb8a20f1a1811ccec632e5faf22156fd547d64079fd789e259aeda5c not found: ID does not exist"
Nov 02 01:55:04.049614 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:04.049489    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg" podStartSLOduration=3.049471973 podStartE2EDuration="3.049471973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:04.045633097 +0000 UTC m=+1768.949315212" watchObservedRunningTime="2026-11-02 01:55:04.049471973 +0000 UTC m=+1768.953154087"
Nov 02 01:55:04.192086 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:04.191599    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"]
Nov 02 01:55:04.238685 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:04.238638    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-lcxx4"]
Nov 02 01:55:04.730268 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:04.730237    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw"
Nov 02 01:55:04.730494 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:04.730355    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg"
Nov 02 01:55:05.221039 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:05.221003    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb" path="/var/lib/kubelet/pods/2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb/volumes"
Nov 02 01:55:13.142988 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:13.142953    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"]
Nov 02 01:55:13.143487 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:13.143151    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz" podUID="47285028-a487-437d-8cd9-63221ffca4d6" containerName="istio-proxy" containerID="cri-o://22dbaa0aa4fc3f86ba306177aaa06e8d827b0b6fe937b505a40ae6b0873de394" gracePeriod=30
Nov 02 01:55:13.193065 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:13.193025    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"]
Nov 02 01:55:13.193338 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:13.193272    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52" podUID="82ab4f70-ac0f-42f7-8571-93b7ef884c77" containerName="istio-proxy" containerID="cri-o://54f4ac87f7e1b5bb9553cfc5f69715dad39b03e6cf1570ac0b30b8513dd85cab" gracePeriod=30
Nov 02 01:55:13.765654 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:13.765615    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"]
Nov 02 01:55:13.765878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:13.765856    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4" podUID="a8596379-fe63-4152-8569-1b16a8f75d70" containerName="istio-proxy" containerID="cri-o://5b268a42d4f06300441305b3b0bdf09472821369b44baf84a48ba67ba57254e9" gracePeriod=30
Nov 02 01:55:14.946018 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:14.945983    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8"
Nov 02 01:55:14.946551 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:14.946090    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg"
Nov 02 01:55:17.637361 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:17.637322    5639 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:55:17.637774 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:17.637381    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52" podUID="82ab4f70-ac0f-42f7-8571-93b7ef884c77" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:55:18.475721 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.475686    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:55:18.524981 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.524954    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:55:18.575151 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.575117    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-data\") pod \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") "
Nov 02 01:55:18.575358 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.575186    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istiod-ca-cert\") pod \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") "
Nov 02 01:55:18.575358 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.575265    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-token\" (UniqueName: \"kubernetes.io/projected/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-token\") pod \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") "
Nov 02 01:55:18.575510 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.575363    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-workload-certs\") pod \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") "
Nov 02 01:55:18.575510 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.575404    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-workload-socket\") pod \"47285028-a487-437d-8cd9-63221ffca4d6\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") "
Nov 02 01:55:18.575510 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.575426    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-data" pod "82ab4f70-ac0f-42f7-8571-93b7ef884c77" (UID: "82ab4f70-ac0f-42f7-8571-93b7ef884c77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:18.575510 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.575448    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-istio-envoy\") pod \"47285028-a487-437d-8cd9-63221ffca4d6\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") "
Nov 02 01:55:18.575510 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.575479    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-workload-certs\") pod \"47285028-a487-437d-8cd9-63221ffca4d6\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") "
Nov 02 01:55:18.575763 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.575527    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-istio-data\") pod \"47285028-a487-437d-8cd9-63221ffca4d6\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") "
Nov 02 01:55:18.575763 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.575576    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-workload-socket\") pod \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") "
Nov 02 01:55:18.575763 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.575622    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-podinfo\") pod \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") "
Nov 02 01:55:18.575763 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.575675    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-workload-certs" pod "82ab4f70-ac0f-42f7-8571-93b7ef884c77" (UID: "82ab4f70-ac0f-42f7-8571-93b7ef884c77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:18.575763 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.575693    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-workload-socket" pod "47285028-a487-437d-8cd9-63221ffca4d6" (UID: "47285028-a487-437d-8cd9-63221ffca4d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:18.575763 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.575707    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istiod-ca-cert" pod "82ab4f70-ac0f-42f7-8571-93b7ef884c77" (UID: "82ab4f70-ac0f-42f7-8571-93b7ef884c77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:55:18.575763 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.575756    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-envoy\") pod \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") "
Nov 02 01:55:18.576115 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.575804    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-credential-socket\") pod \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") "
Nov 02 01:55:18.576115 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.575876    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-workload-socket" pod "82ab4f70-ac0f-42f7-8571-93b7ef884c77" (UID: "82ab4f70-ac0f-42f7-8571-93b7ef884c77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:18.576115 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.575957    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47285028-a487-437d-8cd9-63221ffca4d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47285028-a487-437d-8cd9-63221ffca4d6-istiod-ca-cert\") pod \"47285028-a487-437d-8cd9-63221ffca4d6\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") "
Nov 02 01:55:18.576115 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.575990    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-istio-data" pod "47285028-a487-437d-8cd9-63221ffca4d6" (UID: "47285028-a487-437d-8cd9-63221ffca4d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:18.576115 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.576000    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47285028-a487-437d-8cd9-63221ffca4d6-istio-token\" (UniqueName: \"kubernetes.io/projected/47285028-a487-437d-8cd9-63221ffca4d6-istio-token\") pod \"47285028-a487-437d-8cd9-63221ffca4d6\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") "
Nov 02 01:55:18.576462 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.576124    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-workload-certs" pod "47285028-a487-437d-8cd9-63221ffca4d6" (UID: "47285028-a487-437d-8cd9-63221ffca4d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:18.576462 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.576130    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-credential-socket" pod "82ab4f70-ac0f-42f7-8571-93b7ef884c77" (UID: "82ab4f70-ac0f-42f7-8571-93b7ef884c77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:18.576462 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.576059    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47285028-a487-437d-8cd9-63221ffca4d6-kube-api-access-g7k7s\" (UniqueName: \"kubernetes.io/projected/47285028-a487-437d-8cd9-63221ffca4d6-kube-api-access-g7k7s\") pod \"47285028-a487-437d-8cd9-63221ffca4d6\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") "
Nov 02 01:55:18.576740 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.576645    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47285028-a487-437d-8cd9-63221ffca4d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47285028-a487-437d-8cd9-63221ffca4d6-istio-podinfo\") pod \"47285028-a487-437d-8cd9-63221ffca4d6\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") "
Nov 02 01:55:18.576740 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.576691    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-credential-socket\") pod \"47285028-a487-437d-8cd9-63221ffca4d6\" (UID: \"47285028-a487-437d-8cd9-63221ffca4d6\") "
Nov 02 01:55:18.576850 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.576739    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82ab4f70-ac0f-42f7-8571-93b7ef884c77-kube-api-access-2tcjj\" (UniqueName: \"kubernetes.io/projected/82ab4f70-ac0f-42f7-8571-93b7ef884c77-kube-api-access-2tcjj\") pod \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\" (UID: \"82ab4f70-ac0f-42f7-8571-93b7ef884c77\") "
Nov 02 01:55:18.578041 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.577707    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:18.578041 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.577736    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:18.578041 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.577751    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:18.578041 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.577765    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:18.578041 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.577780    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:18.578041 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.577800    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:18.578041 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.577814    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:18.578041 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.577827    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:18.578563 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.578249    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47285028-a487-437d-8cd9-63221ffca4d6-istiod-ca-cert" pod "47285028-a487-437d-8cd9-63221ffca4d6" (UID: "47285028-a487-437d-8cd9-63221ffca4d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:55:18.578563 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.578414    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-credential-socket" pod "47285028-a487-437d-8cd9-63221ffca4d6" (UID: "47285028-a487-437d-8cd9-63221ffca4d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:18.578715 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.578690    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-istio-envoy" pod "47285028-a487-437d-8cd9-63221ffca4d6" (UID: "47285028-a487-437d-8cd9-63221ffca4d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:18.578990 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.578967    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-envoy" pod "82ab4f70-ac0f-42f7-8571-93b7ef884c77" (UID: "82ab4f70-ac0f-42f7-8571-93b7ef884c77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:18.579069 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.579010    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-token" pod "82ab4f70-ac0f-42f7-8571-93b7ef884c77" (UID: "82ab4f70-ac0f-42f7-8571-93b7ef884c77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:55:18.579069 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.579034    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47285028-a487-437d-8cd9-63221ffca4d6-istio-token" pod "47285028-a487-437d-8cd9-63221ffca4d6" (UID: "47285028-a487-437d-8cd9-63221ffca4d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:55:18.579069 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.579048    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47285028-a487-437d-8cd9-63221ffca4d6-kube-api-access-g7k7s" pod "47285028-a487-437d-8cd9-63221ffca4d6" (UID: "47285028-a487-437d-8cd9-63221ffca4d6"). InnerVolumeSpecName "kube-api-access-g7k7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:55:18.579192 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.579116    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-podinfo" pod "82ab4f70-ac0f-42f7-8571-93b7ef884c77" (UID: "82ab4f70-ac0f-42f7-8571-93b7ef884c77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:55:18.579930 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.579906    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ab4f70-ac0f-42f7-8571-93b7ef884c77-kube-api-access-2tcjj" pod "82ab4f70-ac0f-42f7-8571-93b7ef884c77" (UID: "82ab4f70-ac0f-42f7-8571-93b7ef884c77"). InnerVolumeSpecName "kube-api-access-2tcjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:55:18.580618 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.580602    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47285028-a487-437d-8cd9-63221ffca4d6-istio-podinfo" pod "47285028-a487-437d-8cd9-63221ffca4d6" (UID: "47285028-a487-437d-8cd9-63221ffca4d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:55:18.680006 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.679957    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:18.680006 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.679997    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:18.680006 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.680012    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:18.680622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.680025    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47285028-a487-437d-8cd9-63221ffca4d6-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:18.680622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.680039    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47285028-a487-437d-8cd9-63221ffca4d6-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:18.680622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.680053    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7k7s\" (UniqueName: \"kubernetes.io/projected/47285028-a487-437d-8cd9-63221ffca4d6-kube-api-access-g7k7s\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:18.680622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.680068    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47285028-a487-437d-8cd9-63221ffca4d6-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:18.680622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.680081    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47285028-a487-437d-8cd9-63221ffca4d6-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:18.680622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.680095    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tcjj\" (UniqueName: \"kubernetes.io/projected/82ab4f70-ac0f-42f7-8571-93b7ef884c77-kube-api-access-2tcjj\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:18.680622 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:18.680108    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ab4f70-ac0f-42f7-8571-93b7ef884c77-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:19.095773 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.095744    5639 generic.go:356] "Generic (PLEG): container finished" podID="a8596379-fe63-4152-8569-1b16a8f75d70" containerID="5b268a42d4f06300441305b3b0bdf09472821369b44baf84a48ba67ba57254e9" exitCode=0
Nov 02 01:55:19.095956 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.095813    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4" event={"ID":"a8596379-fe63-4152-8569-1b16a8f75d70","Type":"ContainerDied","Data":"5b268a42d4f06300441305b3b0bdf09472821369b44baf84a48ba67ba57254e9"}
Nov 02 01:55:19.097421 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.097393    5639 generic.go:356] "Generic (PLEG): container finished" podID="82ab4f70-ac0f-42f7-8571-93b7ef884c77" containerID="54f4ac87f7e1b5bb9553cfc5f69715dad39b03e6cf1570ac0b30b8513dd85cab" exitCode=0
Nov 02 01:55:19.097537 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.097459    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"
Nov 02 01:55:19.097537 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.097476    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52" event={"ID":"82ab4f70-ac0f-42f7-8571-93b7ef884c77","Type":"ContainerDied","Data":"54f4ac87f7e1b5bb9553cfc5f69715dad39b03e6cf1570ac0b30b8513dd85cab"}
Nov 02 01:55:19.097537 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.097521    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52" event={"ID":"82ab4f70-ac0f-42f7-8571-93b7ef884c77","Type":"ContainerDied","Data":"853e4d4c1b9023cb1f961c9e508c55621cd288422f31b2c9622af0116995ad1e"}
Nov 02 01:55:19.097736 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.097550    5639 scope.go:122] "RemoveContainer" containerID="54f4ac87f7e1b5bb9553cfc5f69715dad39b03e6cf1570ac0b30b8513dd85cab"
Nov 02 01:55:19.099057 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.099027    5639 generic.go:356] "Generic (PLEG): container finished" podID="47285028-a487-437d-8cd9-63221ffca4d6" containerID="22dbaa0aa4fc3f86ba306177aaa06e8d827b0b6fe937b505a40ae6b0873de394" exitCode=0
Nov 02 01:55:19.099352 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.099074    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"
Nov 02 01:55:19.099474 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.099444    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz" event={"ID":"47285028-a487-437d-8cd9-63221ffca4d6","Type":"ContainerDied","Data":"22dbaa0aa4fc3f86ba306177aaa06e8d827b0b6fe937b505a40ae6b0873de394"}
Nov 02 01:55:19.099593 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.099506    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz" event={"ID":"47285028-a487-437d-8cd9-63221ffca4d6","Type":"ContainerDied","Data":"f6af77db99e1f05f4bcc4930d952341303ac736390ac76833dda9c69238a8599"}
Nov 02 01:55:19.109977 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.109958    5639 scope.go:122] "RemoveContainer" containerID="54f4ac87f7e1b5bb9553cfc5f69715dad39b03e6cf1570ac0b30b8513dd85cab"
Nov 02 01:55:19.110237 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:55:19.110218    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54f4ac87f7e1b5bb9553cfc5f69715dad39b03e6cf1570ac0b30b8513dd85cab\": container with ID starting with 54f4ac87f7e1b5bb9553cfc5f69715dad39b03e6cf1570ac0b30b8513dd85cab not found: ID does not exist" containerID="54f4ac87f7e1b5bb9553cfc5f69715dad39b03e6cf1570ac0b30b8513dd85cab"
Nov 02 01:55:19.110311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.110243    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54f4ac87f7e1b5bb9553cfc5f69715dad39b03e6cf1570ac0b30b8513dd85cab"} err="failed to get container status \"54f4ac87f7e1b5bb9553cfc5f69715dad39b03e6cf1570ac0b30b8513dd85cab\": rpc error: code = NotFound desc = could not find container \"54f4ac87f7e1b5bb9553cfc5f69715dad39b03e6cf1570ac0b30b8513dd85cab\": container with ID starting with 54f4ac87f7e1b5bb9553cfc5f69715dad39b03e6cf1570ac0b30b8513dd85cab not found: ID does not exist"
Nov 02 01:55:19.110311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.110259    5639 scope.go:122] "RemoveContainer" containerID="22dbaa0aa4fc3f86ba306177aaa06e8d827b0b6fe937b505a40ae6b0873de394"
Nov 02 01:55:19.120240 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.120222    5639 scope.go:122] "RemoveContainer" containerID="22dbaa0aa4fc3f86ba306177aaa06e8d827b0b6fe937b505a40ae6b0873de394"
Nov 02 01:55:19.120557 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:55:19.120538    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22dbaa0aa4fc3f86ba306177aaa06e8d827b0b6fe937b505a40ae6b0873de394\": container with ID starting with 22dbaa0aa4fc3f86ba306177aaa06e8d827b0b6fe937b505a40ae6b0873de394 not found: ID does not exist" containerID="22dbaa0aa4fc3f86ba306177aaa06e8d827b0b6fe937b505a40ae6b0873de394"
Nov 02 01:55:19.120602 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.120564    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22dbaa0aa4fc3f86ba306177aaa06e8d827b0b6fe937b505a40ae6b0873de394"} err="failed to get container status \"22dbaa0aa4fc3f86ba306177aaa06e8d827b0b6fe937b505a40ae6b0873de394\": rpc error: code = NotFound desc = could not find container \"22dbaa0aa4fc3f86ba306177aaa06e8d827b0b6fe937b505a40ae6b0873de394\": container with ID starting with 22dbaa0aa4fc3f86ba306177aaa06e8d827b0b6fe937b505a40ae6b0873de394 not found: ID does not exist"
Nov 02 01:55:19.263188 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.263154    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"]
Nov 02 01:55:19.312244 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.312214    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-bvr52"]
Nov 02 01:55:19.380610 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.380548    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"]
Nov 02 01:55:19.384322 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.384294    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-78pvz"]
Nov 02 01:55:19.458557 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.458524    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:55:19.614360 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.601304    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-workload-socket\") pod \"a8596379-fe63-4152-8569-1b16a8f75d70\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") "
Nov 02 01:55:19.614360 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.601509    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-workload-certs\") pod \"a8596379-fe63-4152-8569-1b16a8f75d70\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") "
Nov 02 01:55:19.614360 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.601583    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8596379-fe63-4152-8569-1b16a8f75d70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8596379-fe63-4152-8569-1b16a8f75d70-istio-podinfo\") pod \"a8596379-fe63-4152-8569-1b16a8f75d70\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") "
Nov 02 01:55:19.614360 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.601623    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8596379-fe63-4152-8569-1b16a8f75d70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8596379-fe63-4152-8569-1b16a8f75d70-istiod-ca-cert\") pod \"a8596379-fe63-4152-8569-1b16a8f75d70\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") "
Nov 02 01:55:19.614360 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.601689    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-istio-data\") pod \"a8596379-fe63-4152-8569-1b16a8f75d70\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") "
Nov 02 01:55:19.614360 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.601841    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8596379-fe63-4152-8569-1b16a8f75d70-kube-api-access-d29zf\" (UniqueName: \"kubernetes.io/projected/a8596379-fe63-4152-8569-1b16a8f75d70-kube-api-access-d29zf\") pod \"a8596379-fe63-4152-8569-1b16a8f75d70\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") "
Nov 02 01:55:19.614360 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.601905    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-istio-envoy\") pod \"a8596379-fe63-4152-8569-1b16a8f75d70\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") "
Nov 02 01:55:19.614360 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.601965    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8596379-fe63-4152-8569-1b16a8f75d70-istio-token\" (UniqueName: \"kubernetes.io/projected/a8596379-fe63-4152-8569-1b16a8f75d70-istio-token\") pod \"a8596379-fe63-4152-8569-1b16a8f75d70\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") "
Nov 02 01:55:19.614360 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.601995    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-credential-socket\") pod \"a8596379-fe63-4152-8569-1b16a8f75d70\" (UID: \"a8596379-fe63-4152-8569-1b16a8f75d70\") "
Nov 02 01:55:19.614360 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.608738    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8596379-fe63-4152-8569-1b16a8f75d70-istiod-ca-cert" pod "a8596379-fe63-4152-8569-1b16a8f75d70" (UID: "a8596379-fe63-4152-8569-1b16a8f75d70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:55:19.614360 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.608886    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-workload-socket" pod "a8596379-fe63-4152-8569-1b16a8f75d70" (UID: "a8596379-fe63-4152-8569-1b16a8f75d70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:19.614360 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.609011    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-workload-certs" pod "a8596379-fe63-4152-8569-1b16a8f75d70" (UID: "a8596379-fe63-4152-8569-1b16a8f75d70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:19.617552 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.617263    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-istio-data" pod "a8596379-fe63-4152-8569-1b16a8f75d70" (UID: "a8596379-fe63-4152-8569-1b16a8f75d70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:19.622418 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.619439    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-credential-socket" pod "a8596379-fe63-4152-8569-1b16a8f75d70" (UID: "a8596379-fe63-4152-8569-1b16a8f75d70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:19.644548 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.644486    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8596379-fe63-4152-8569-1b16a8f75d70-istio-token" pod "a8596379-fe63-4152-8569-1b16a8f75d70" (UID: "a8596379-fe63-4152-8569-1b16a8f75d70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:55:19.648940 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.648425    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-istio-envoy" pod "a8596379-fe63-4152-8569-1b16a8f75d70" (UID: "a8596379-fe63-4152-8569-1b16a8f75d70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:19.656379 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.653721    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8596379-fe63-4152-8569-1b16a8f75d70-kube-api-access-d29zf" pod "a8596379-fe63-4152-8569-1b16a8f75d70" (UID: "a8596379-fe63-4152-8569-1b16a8f75d70"). InnerVolumeSpecName "kube-api-access-d29zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:55:19.663494 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.663435    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8596379-fe63-4152-8569-1b16a8f75d70-istio-podinfo" pod "a8596379-fe63-4152-8569-1b16a8f75d70" (UID: "a8596379-fe63-4152-8569-1b16a8f75d70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:55:19.783637 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.783610    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:19.784210 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.784189    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8596379-fe63-4152-8569-1b16a8f75d70-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:19.784365 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.784350    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8596379-fe63-4152-8569-1b16a8f75d70-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:19.784478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.784466    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:19.784678 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.784660    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d29zf\" (UniqueName: \"kubernetes.io/projected/a8596379-fe63-4152-8569-1b16a8f75d70-kube-api-access-d29zf\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:19.784789 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.784775    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:19.784905 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.784890    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8596379-fe63-4152-8569-1b16a8f75d70-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:19.784997 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.784985    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:19.785095 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:19.785079    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8596379-fe63-4152-8569-1b16a8f75d70-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:20.109469 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:20.109431    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"
Nov 02 01:55:20.109675 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:20.109430    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4" event={"ID":"a8596379-fe63-4152-8569-1b16a8f75d70","Type":"ContainerDied","Data":"cf588bc4b324eafa0716d2b55cdd067b043dae4aaf0be3ed0039f04fa36a9e53"}
Nov 02 01:55:20.109675 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:20.109575    5639 scope.go:122] "RemoveContainer" containerID="5b268a42d4f06300441305b3b0bdf09472821369b44baf84a48ba67ba57254e9"
Nov 02 01:55:20.276416 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:20.276369    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"]
Nov 02 01:55:20.326218 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:20.326184    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-cqdq4"]
Nov 02 01:55:20.946362 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:20.946327    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"]
Nov 02 01:55:20.947040 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:20.947024    5639 cpu_manager.go:414] "containerMap: removing container" podUID="47285028-a487-437d-8cd9-63221ffca4d6" containerName="istio-proxy"
Nov 02 01:55:20.947081 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:20.947041    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47285028-a487-437d-8cd9-63221ffca4d6" containerName="istio-proxy"
Nov 02 01:55:20.947081 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:20.947057    5639 cpu_manager.go:414] "containerMap: removing container" podUID="82ab4f70-ac0f-42f7-8571-93b7ef884c77" containerName="istio-proxy"
Nov 02 01:55:20.947081 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:20.947062    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82ab4f70-ac0f-42f7-8571-93b7ef884c77" containerName="istio-proxy"
Nov 02 01:55:20.947172 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:20.947088    5639 cpu_manager.go:414] "containerMap: removing container" podUID="a8596379-fe63-4152-8569-1b16a8f75d70" containerName="istio-proxy"
Nov 02 01:55:20.947172 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:20.947093    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8596379-fe63-4152-8569-1b16a8f75d70" containerName="istio-proxy"
Nov 02 01:55:20.947172 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:20.947106    5639 cpu_manager.go:414] "containerMap: removing container" podUID="2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb" containerName="istio-proxy"
Nov 02 01:55:20.947172 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:20.947111    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb" containerName="istio-proxy"
Nov 02 01:55:20.947319 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:20.947194    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8596379-fe63-4152-8569-1b16a8f75d70" containerName="istio-proxy"
Nov 02 01:55:20.947319 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:20.947204    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c5686ca-e1a1-4cbd-bd35-767ee2ae40fb" containerName="istio-proxy"
Nov 02 01:55:20.947319 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:20.947215    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="47285028-a487-437d-8cd9-63221ffca4d6" containerName="istio-proxy"
Nov 02 01:55:20.947319 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:20.947222    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="82ab4f70-ac0f-42f7-8571-93b7ef884c77" containerName="istio-proxy"
Nov 02 01:55:20.952247 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:20.952227    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.009384 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.008559    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-218802-openshift-default-dockercfg-scljq\""
Nov 02 01:55:21.016430 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.015039    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"]
Nov 02 01:55:21.019872 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.019836    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-workload-socket\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.020035 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.020019    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-istio-data\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.020103 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.020091    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xdcp\" (UniqueName: \"kubernetes.io/projected/7542d266-7f8c-474a-a8dc-9848376e3993-kube-api-access-2xdcp\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.020194 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.020177    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-istio-envoy\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.022579 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.022546    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7542d266-7f8c-474a-a8dc-9848376e3993-istio-token\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.022696 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.022642    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-workload-certs\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.022740 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.022700    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7542d266-7f8c-474a-a8dc-9848376e3993-istio-podinfo\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.022778 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.022762    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-credential-socket\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.022814 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.022802    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7542d266-7f8c-474a-a8dc-9848376e3993-istiod-ca-cert\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.199367 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.194336    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-istio-data\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.199367 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.194426    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xdcp\" (UniqueName: \"kubernetes.io/projected/7542d266-7f8c-474a-a8dc-9848376e3993-kube-api-access-2xdcp\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.199367 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.194467    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-istio-envoy\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.199367 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.194556    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7542d266-7f8c-474a-a8dc-9848376e3993-istio-token\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.199367 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.194611    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-workload-certs\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.199367 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.194662    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7542d266-7f8c-474a-a8dc-9848376e3993-istio-podinfo\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.199367 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.194729    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-credential-socket\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.199367 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.194759    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7542d266-7f8c-474a-a8dc-9848376e3993-istiod-ca-cert\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.199367 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.194883    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-workload-socket\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.233304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.231002    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-workload-socket\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.260303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.250954    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7542d266-7f8c-474a-a8dc-9848376e3993-istio-podinfo\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.337822 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.332625    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-istio-data\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.337822 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.336541    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-credential-socket\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.337822 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.336791    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-workload-certs\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.337822 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.337203    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7542d266-7f8c-474a-a8dc-9848376e3993-istiod-ca-cert\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.402152 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.400366    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xdcp\" (UniqueName: \"kubernetes.io/projected/7542d266-7f8c-474a-a8dc-9848376e3993-kube-api-access-2xdcp\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.402152 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.400634    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-istio-envoy\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.404246 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.404188    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47285028-a487-437d-8cd9-63221ffca4d6" path="/var/lib/kubelet/pods/47285028-a487-437d-8cd9-63221ffca4d6/volumes"
Nov 02 01:55:21.405139 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.405091    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82ab4f70-ac0f-42f7-8571-93b7ef884c77" path="/var/lib/kubelet/pods/82ab4f70-ac0f-42f7-8571-93b7ef884c77/volumes"
Nov 02 01:55:21.406061 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.405984    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7542d266-7f8c-474a-a8dc-9848376e3993-istio-token\") pod \"e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") " pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:21.428065 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.428024    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8596379-fe63-4152-8569-1b16a8f75d70" path="/var/lib/kubelet/pods/a8596379-fe63-4152-8569-1b16a8f75d70/volumes"
Nov 02 01:55:21.604550 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:21.604518    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:22.037976 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:22.037952    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"]
Nov 02 01:55:22.038746 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:55:22.038717    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7542d266_7f8c_474a_a8dc_9848376e3993.slice/crio-0ce62ce0ffb5367e94cdbc3dd41d5032fd1eb1dbbe68dba4fe3681349b224cf1 WatchSource:0}: Error finding container 0ce62ce0ffb5367e94cdbc3dd41d5032fd1eb1dbbe68dba4fe3681349b224cf1: Status 404 returned error can't find the container with id 0ce62ce0ffb5367e94cdbc3dd41d5032fd1eb1dbbe68dba4fe3681349b224cf1
Nov 02 01:55:22.041727 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:22.041689    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:55:22.041809 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:22.041774    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:55:22.041856 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:22.041814    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:55:22.241132 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:22.241100    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs" event={"ID":"7542d266-7f8c-474a-a8dc-9848376e3993","Type":"ContainerStarted","Data":"e662c189e08dcec93fdbe7cc8b5a01617477d671284cbf448c38a4df9f102569"}
Nov 02 01:55:22.241328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:22.241153    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs" event={"ID":"7542d266-7f8c-474a-a8dc-9848376e3993","Type":"ContainerStarted","Data":"0ce62ce0ffb5367e94cdbc3dd41d5032fd1eb1dbbe68dba4fe3681349b224cf1"}
Nov 02 01:55:22.361743 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:22.361673    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs" podStartSLOduration=2.361659847 podStartE2EDuration="2.361659847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:22.360568567 +0000 UTC m=+1787.264250683" watchObservedRunningTime="2026-11-02 01:55:22.361659847 +0000 UTC m=+1787.265342024"
Nov 02 01:55:22.604788 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:22.604752    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:23.611548 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:23.611515    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:24.256410 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:24.256383    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:24.257356 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:24.257334    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:55:25.331305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.330577    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"]
Nov 02 01:55:25.362318 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.341165    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.418313 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.403450    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"]
Nov 02 01:55:25.420599 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.420552    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-istio-envoy\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.420845 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.420824    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-workload-certs\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.420993 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.420977    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-credential-socket\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.421364 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.421344    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/124b9d97-7aec-44dd-a24b-06be952004fc-istiod-ca-cert\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.421501 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.421485    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-istio-data\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.464427 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.462710    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/124b9d97-7aec-44dd-a24b-06be952004fc-istio-token\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.464884 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.432213    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-bb0464-openshift-default-dockercfg-4ns7g\""
Nov 02 01:55:25.465189 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.465150    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsz48\" (UniqueName: \"kubernetes.io/projected/124b9d97-7aec-44dd-a24b-06be952004fc-kube-api-access-jsz48\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.466042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.466002    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-workload-socket\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.466348 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.466329    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/124b9d97-7aec-44dd-a24b-06be952004fc-istio-podinfo\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.568029 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.567995    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsz48\" (UniqueName: \"kubernetes.io/projected/124b9d97-7aec-44dd-a24b-06be952004fc-kube-api-access-jsz48\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.568365 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.568110    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-workload-socket\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.568365 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.568165    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/124b9d97-7aec-44dd-a24b-06be952004fc-istio-podinfo\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.568365 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.568206    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-istio-envoy\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.568365 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.568254    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-workload-certs\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.568365 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.568338    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-credential-socket\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.568881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.568416    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/124b9d97-7aec-44dd-a24b-06be952004fc-istiod-ca-cert\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.568881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.568453    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-istio-data\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.568881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.568492    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/124b9d97-7aec-44dd-a24b-06be952004fc-istio-token\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.568881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.568618    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-workload-socket\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.568881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.568674    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-workload-certs\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.569224 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.568982    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-credential-socket\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.569224 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.569005    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-istio-data\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.569375 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.569265    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/124b9d97-7aec-44dd-a24b-06be952004fc-istiod-ca-cert\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.571792 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.571751    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-istio-envoy\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.571921 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.571796    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/124b9d97-7aec-44dd-a24b-06be952004fc-istio-podinfo\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.633316 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.633215    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/124b9d97-7aec-44dd-a24b-06be952004fc-istio-token\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.633488 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.633348    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsz48\" (UniqueName: \"kubernetes.io/projected/124b9d97-7aec-44dd-a24b-06be952004fc-kube-api-access-jsz48\") pod \"e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") " pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:25.678476 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:25.678440    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:26.505572 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:26.505548    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"]
Nov 02 01:55:26.507756 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:55:26.507728    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124b9d97_7aec_44dd_a24b_06be952004fc.slice/crio-e2e70ca3ab11f53147a9cd6913782c8a2ce0a3cde20b64fa9952a3987ac7fade WatchSource:0}: Error finding container e2e70ca3ab11f53147a9cd6913782c8a2ce0a3cde20b64fa9952a3987ac7fade: Status 404 returned error can't find the container with id e2e70ca3ab11f53147a9cd6913782c8a2ce0a3cde20b64fa9952a3987ac7fade
Nov 02 01:55:26.510591 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:26.510559    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:55:26.510658 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:26.510628    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:55:26.510691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:26.510657    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:55:27.316332 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:27.316298    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd" event={"ID":"124b9d97-7aec-44dd-a24b-06be952004fc","Type":"ContainerStarted","Data":"a599f47014a914bad2b2f716f29a0bf30b66f348c4890665f5f2ace86edea778"}
Nov 02 01:55:27.316513 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:27.316355    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd" event={"ID":"124b9d97-7aec-44dd-a24b-06be952004fc","Type":"ContainerStarted","Data":"e2e70ca3ab11f53147a9cd6913782c8a2ce0a3cde20b64fa9952a3987ac7fade"}
Nov 02 01:55:27.422694 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:55:27.422658    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:55:27.438713 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:27.438659    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd" podStartSLOduration=2.438644003 podStartE2EDuration="2.438644003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:27.43603734 +0000 UTC m=+1792.339719454" watchObservedRunningTime="2026-11-02 01:55:27.438644003 +0000 UTC m=+1792.342326117"
Nov 02 01:55:27.679260 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:27.679226    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:27.684902 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:27.684876    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:28.324815 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:28.324782    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:28.325635 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:28.325611    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:55:29.822930 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:29.822896    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"]
Nov 02 01:55:29.823443 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:29.823137    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs" podUID="f975458f-bee6-46a6-a92c-189a9509d0e6" containerName="istio-proxy" containerID="cri-o://e984a9f7100c676ef95d95d14ad56126bb00e33caf4aa4ea6c3d3bffb3a6092c" gracePeriod=30
Nov 02 01:55:32.354903 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:32.354866    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf"]
Nov 02 01:55:32.359812 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:32.359786    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf"
Nov 02 01:55:32.416958 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:32.416926    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf"]
Nov 02 01:55:32.468607 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:32.468565    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-a7c06d-dockercfg-ckd72\""
Nov 02 01:55:32.474645 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:32.469022    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-a7c06d\""
Nov 02 01:55:32.726059 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:32.725658    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l92r\" (UniqueName: \"kubernetes.io/projected/1c1d68fb-24ac-454b-bf9c-02a894c43862-kube-api-access-9l92r\") pod \"payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf\" (UID: \"1c1d68fb-24ac-454b-bf9c-02a894c43862\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf"
Nov 02 01:55:32.726059 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:32.725699    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c1d68fb-24ac-454b-bf9c-02a894c43862-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf\" (UID: \"1c1d68fb-24ac-454b-bf9c-02a894c43862\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf"
Nov 02 01:55:32.743435 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:32.743400    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r"]
Nov 02 01:55:32.749317 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:32.749268    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r"]
Nov 02 01:55:32.749563 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:32.749390    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r"
Nov 02 01:55:32.826742 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:32.826704    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l92r\" (UniqueName: \"kubernetes.io/projected/1c1d68fb-24ac-454b-bf9c-02a894c43862-kube-api-access-9l92r\") pod \"payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf\" (UID: \"1c1d68fb-24ac-454b-bf9c-02a894c43862\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf"
Nov 02 01:55:32.826974 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:32.826754    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c1d68fb-24ac-454b-bf9c-02a894c43862-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf\" (UID: \"1c1d68fb-24ac-454b-bf9c-02a894c43862\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf"
Nov 02 01:55:32.826974 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:32.826886    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98bf5ee6-a486-456d-b9d8-d79bbaf5717e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r\" (UID: \"98bf5ee6-a486-456d-b9d8-d79bbaf5717e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r"
Nov 02 01:55:32.826974 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:32.826954    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2qsr\" (UniqueName: \"kubernetes.io/projected/98bf5ee6-a486-456d-b9d8-d79bbaf5717e-kube-api-access-v2qsr\") pod \"payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r\" (UID: \"98bf5ee6-a486-456d-b9d8-d79bbaf5717e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r"
Nov 02 01:55:32.827744 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:32.827714    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c1d68fb-24ac-454b-bf9c-02a894c43862-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf\" (UID: \"1c1d68fb-24ac-454b-bf9c-02a894c43862\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf"
Nov 02 01:55:32.888396 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:32.888364    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l92r\" (UniqueName: \"kubernetes.io/projected/1c1d68fb-24ac-454b-bf9c-02a894c43862-kube-api-access-9l92r\") pod \"payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf\" (UID: \"1c1d68fb-24ac-454b-bf9c-02a894c43862\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf"
Nov 02 01:55:32.928609 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:32.928565    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98bf5ee6-a486-456d-b9d8-d79bbaf5717e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r\" (UID: \"98bf5ee6-a486-456d-b9d8-d79bbaf5717e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r"
Nov 02 01:55:32.928824 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:32.928670    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2qsr\" (UniqueName: \"kubernetes.io/projected/98bf5ee6-a486-456d-b9d8-d79bbaf5717e-kube-api-access-v2qsr\") pod \"payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r\" (UID: \"98bf5ee6-a486-456d-b9d8-d79bbaf5717e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r"
Nov 02 01:55:32.929258 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:32.929237    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98bf5ee6-a486-456d-b9d8-d79bbaf5717e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r\" (UID: \"98bf5ee6-a486-456d-b9d8-d79bbaf5717e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r"
Nov 02 01:55:32.975823 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:32.975789    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf"
Nov 02 01:55:32.996828 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:32.996800    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2qsr\" (UniqueName: \"kubernetes.io/projected/98bf5ee6-a486-456d-b9d8-d79bbaf5717e-kube-api-access-v2qsr\") pod \"payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r\" (UID: \"98bf5ee6-a486-456d-b9d8-d79bbaf5717e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r"
Nov 02 01:55:33.065751 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:33.065716    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r"
Nov 02 01:55:33.358946 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:33.358918    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf"]
Nov 02 01:55:33.359882 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:55:33.359860    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c1d68fb_24ac_454b_bf9c_02a894c43862.slice/crio-3f7faff5c2043534aec9afcdcc3d0a999cb6d8718bff239fa8b6e195b7e8be97 WatchSource:0}: Error finding container 3f7faff5c2043534aec9afcdcc3d0a999cb6d8718bff239fa8b6e195b7e8be97: Status 404 returned error can't find the container with id 3f7faff5c2043534aec9afcdcc3d0a999cb6d8718bff239fa8b6e195b7e8be97
Nov 02 01:55:33.398243 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:33.398205    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf" event={"ID":"1c1d68fb-24ac-454b-bf9c-02a894c43862","Type":"ContainerStarted","Data":"3f7faff5c2043534aec9afcdcc3d0a999cb6d8718bff239fa8b6e195b7e8be97"}
Nov 02 01:55:33.469756 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:33.469725    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r"]
Nov 02 01:55:33.471232 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:55:33.471205    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98bf5ee6_a486_456d_b9d8_d79bbaf5717e.slice/crio-9de058b74e8e9649c392c480730adf6af47cfba3629763721b789698ab5898b0 WatchSource:0}: Error finding container 9de058b74e8e9649c392c480730adf6af47cfba3629763721b789698ab5898b0: Status 404 returned error can't find the container with id 9de058b74e8e9649c392c480730adf6af47cfba3629763721b789698ab5898b0
Nov 02 01:55:34.407506 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:34.407466    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r" event={"ID":"98bf5ee6-a486-456d-b9d8-d79bbaf5717e","Type":"ContainerStarted","Data":"919752363bc6a1ce089d0858dbc15cf073b144a55b23d3228ce1a781e74141e8"}
Nov 02 01:55:34.407945 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:34.407532    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r" event={"ID":"98bf5ee6-a486-456d-b9d8-d79bbaf5717e","Type":"ContainerStarted","Data":"9de058b74e8e9649c392c480730adf6af47cfba3629763721b789698ab5898b0"}
Nov 02 01:55:34.407945 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:34.407565    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r"
Nov 02 01:55:34.409309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:34.409251    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf" event={"ID":"1c1d68fb-24ac-454b-bf9c-02a894c43862","Type":"ContainerStarted","Data":"767d814c712ed1734f9e15a5792b601ec03c702557cdb508184d6ce3fc255e2b"}
Nov 02 01:55:34.409452 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:34.409387    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf"
Nov 02 01:55:34.524188 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:34.524118    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r" podStartSLOduration=2.524100279 podStartE2EDuration="2.524100279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:34.522543645 +0000 UTC m=+1799.426225759" watchObservedRunningTime="2026-11-02 01:55:34.524100279 +0000 UTC m=+1799.427782394"
Nov 02 01:55:34.665466 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:34.649183    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf" podStartSLOduration=2.64916251 podStartE2EDuration="2.64916251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:34.646945604 +0000 UTC m=+1799.550627720" watchObservedRunningTime="2026-11-02 01:55:34.64916251 +0000 UTC m=+1799.552844625"
Nov 02 01:55:35.100168 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.100143    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:55:35.182097 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.182065    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-data\") pod \"f975458f-bee6-46a6-a92c-189a9509d0e6\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") "
Nov 02 01:55:35.182261 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.182147    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-podinfo\") pod \"f975458f-bee6-46a6-a92c-189a9509d0e6\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") "
Nov 02 01:55:35.182261 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.182245    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-envoy\") pod \"f975458f-bee6-46a6-a92c-189a9509d0e6\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") "
Nov 02 01:55:35.182428 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.182274    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-credential-socket\") pod \"f975458f-bee6-46a6-a92c-189a9509d0e6\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") "
Nov 02 01:55:35.182428 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.182389    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-workload-certs\") pod \"f975458f-bee6-46a6-a92c-189a9509d0e6\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") "
Nov 02 01:55:35.182428 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.182392    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-data" pod "f975458f-bee6-46a6-a92c-189a9509d0e6" (UID: "f975458f-bee6-46a6-a92c-189a9509d0e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:35.182585 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.182476    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f975458f-bee6-46a6-a92c-189a9509d0e6-kube-api-access-28vng\" (UniqueName: \"kubernetes.io/projected/f975458f-bee6-46a6-a92c-189a9509d0e6-kube-api-access-28vng\") pod \"f975458f-bee6-46a6-a92c-189a9509d0e6\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") "
Nov 02 01:55:35.182585 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.182504    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-workload-socket\") pod \"f975458f-bee6-46a6-a92c-189a9509d0e6\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") "
Nov 02 01:55:35.182585 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.182582    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-token\" (UniqueName: \"kubernetes.io/projected/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-token\") pod \"f975458f-bee6-46a6-a92c-189a9509d0e6\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") "
Nov 02 01:55:35.182754 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.182691    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f975458f-bee6-46a6-a92c-189a9509d0e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f975458f-bee6-46a6-a92c-189a9509d0e6-istiod-ca-cert\") pod \"f975458f-bee6-46a6-a92c-189a9509d0e6\" (UID: \"f975458f-bee6-46a6-a92c-189a9509d0e6\") "
Nov 02 01:55:35.182812 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.182753    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-workload-certs" pod "f975458f-bee6-46a6-a92c-189a9509d0e6" (UID: "f975458f-bee6-46a6-a92c-189a9509d0e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:35.182812 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.182771    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-credential-socket" pod "f975458f-bee6-46a6-a92c-189a9509d0e6" (UID: "f975458f-bee6-46a6-a92c-189a9509d0e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:35.183171 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.183146    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-workload-socket" pod "f975458f-bee6-46a6-a92c-189a9509d0e6" (UID: "f975458f-bee6-46a6-a92c-189a9509d0e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:35.183476 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.183397    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:35.183476 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.183412    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:35.183476 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.183423    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:35.183476 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.183431    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:35.184057 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.184027    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f975458f-bee6-46a6-a92c-189a9509d0e6-istiod-ca-cert" pod "f975458f-bee6-46a6-a92c-189a9509d0e6" (UID: "f975458f-bee6-46a6-a92c-189a9509d0e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:55:35.184659 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.184634    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-podinfo" pod "f975458f-bee6-46a6-a92c-189a9509d0e6" (UID: "f975458f-bee6-46a6-a92c-189a9509d0e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:55:35.184981 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.184962    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-envoy" pod "f975458f-bee6-46a6-a92c-189a9509d0e6" (UID: "f975458f-bee6-46a6-a92c-189a9509d0e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:35.185187 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.185128    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-token" pod "f975458f-bee6-46a6-a92c-189a9509d0e6" (UID: "f975458f-bee6-46a6-a92c-189a9509d0e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:55:35.185501 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.185480    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f975458f-bee6-46a6-a92c-189a9509d0e6-kube-api-access-28vng" pod "f975458f-bee6-46a6-a92c-189a9509d0e6" (UID: "f975458f-bee6-46a6-a92c-189a9509d0e6"). InnerVolumeSpecName "kube-api-access-28vng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:55:35.285069 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.285023    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:35.285069 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.285070    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:35.285308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.285087    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28vng\" (UniqueName: \"kubernetes.io/projected/f975458f-bee6-46a6-a92c-189a9509d0e6-kube-api-access-28vng\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:35.285308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.285102    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f975458f-bee6-46a6-a92c-189a9509d0e6-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:35.285308 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.285117    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f975458f-bee6-46a6-a92c-189a9509d0e6-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:35.418733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.418702    5639 generic.go:356] "Generic (PLEG): container finished" podID="f975458f-bee6-46a6-a92c-189a9509d0e6" containerID="e984a9f7100c676ef95d95d14ad56126bb00e33caf4aa4ea6c3d3bffb3a6092c" exitCode=0
Nov 02 01:55:35.419182 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.418780    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"
Nov 02 01:55:35.419182 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.418784    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs" event={"ID":"f975458f-bee6-46a6-a92c-189a9509d0e6","Type":"ContainerDied","Data":"e984a9f7100c676ef95d95d14ad56126bb00e33caf4aa4ea6c3d3bffb3a6092c"}
Nov 02 01:55:35.419182 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.418842    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs" event={"ID":"f975458f-bee6-46a6-a92c-189a9509d0e6","Type":"ContainerDied","Data":"4061b44f589099da90ab76727fc52879a73710f14d391bbf0c121561bccbc0f5"}
Nov 02 01:55:35.419182 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.418864    5639 scope.go:122] "RemoveContainer" containerID="e984a9f7100c676ef95d95d14ad56126bb00e33caf4aa4ea6c3d3bffb3a6092c"
Nov 02 01:55:35.430057 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.430034    5639 scope.go:122] "RemoveContainer" containerID="e984a9f7100c676ef95d95d14ad56126bb00e33caf4aa4ea6c3d3bffb3a6092c"
Nov 02 01:55:35.430427 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:55:35.430406    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e984a9f7100c676ef95d95d14ad56126bb00e33caf4aa4ea6c3d3bffb3a6092c\": container with ID starting with e984a9f7100c676ef95d95d14ad56126bb00e33caf4aa4ea6c3d3bffb3a6092c not found: ID does not exist" containerID="e984a9f7100c676ef95d95d14ad56126bb00e33caf4aa4ea6c3d3bffb3a6092c"
Nov 02 01:55:35.430511 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.430436    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e984a9f7100c676ef95d95d14ad56126bb00e33caf4aa4ea6c3d3bffb3a6092c"} err="failed to get container status \"e984a9f7100c676ef95d95d14ad56126bb00e33caf4aa4ea6c3d3bffb3a6092c\": rpc error: code = NotFound desc = could not find container \"e984a9f7100c676ef95d95d14ad56126bb00e33caf4aa4ea6c3d3bffb3a6092c\": container with ID starting with e984a9f7100c676ef95d95d14ad56126bb00e33caf4aa4ea6c3d3bffb3a6092c not found: ID does not exist"
Nov 02 01:55:35.585721 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.585684    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"]
Nov 02 01:55:35.589046 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:35.589024    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-679fs"]
Nov 02 01:55:36.480072 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:36.480037    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fce56afe-g94dz"]
Nov 02 01:55:36.480758 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:36.480742    5639 cpu_manager.go:414] "containerMap: removing container" podUID="f975458f-bee6-46a6-a92c-189a9509d0e6" containerName="istio-proxy"
Nov 02 01:55:36.480807 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:36.480758    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f975458f-bee6-46a6-a92c-189a9509d0e6" containerName="istio-proxy"
Nov 02 01:55:36.480906 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:36.480897    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="f975458f-bee6-46a6-a92c-189a9509d0e6" containerName="istio-proxy"
Nov 02 01:55:36.486092 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:36.486072    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce56afe-g94dz"
Nov 02 01:55:36.538566 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:36.538515    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:55:36.538747 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:36.538609    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:55:36.542468 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:36.542445    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fce56afe-g94dz"]
Nov 02 01:55:36.586994 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:36.586963    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:55:36.587150 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:36.587013    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:55:36.604002 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:36.603971    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3066172f-0a14-486e-9cba-a2177d0760ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fce56afe-g94dz\" (UID: \"3066172f-0a14-486e-9cba-a2177d0760ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce56afe-g94dz"
Nov 02 01:55:36.604205 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:36.604030    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3066172f-0a14-486e-9cba-a2177d0760ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fce56afe-g94dz\" (UID: \"3066172f-0a14-486e-9cba-a2177d0760ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce56afe-g94dz"
Nov 02 01:55:36.604264 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:36.604221    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtswz\" (UniqueName: \"kubernetes.io/projected/3066172f-0a14-486e-9cba-a2177d0760ce-kube-api-access-jtswz\") pod \"maas-api-revoke-sub-fce56afe-g94dz\" (UID: \"3066172f-0a14-486e-9cba-a2177d0760ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce56afe-g94dz"
Nov 02 01:55:36.714458 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:36.708340    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3066172f-0a14-486e-9cba-a2177d0760ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fce56afe-g94dz\" (UID: \"3066172f-0a14-486e-9cba-a2177d0760ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce56afe-g94dz"
Nov 02 01:55:36.714768 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:36.714741    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3066172f-0a14-486e-9cba-a2177d0760ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fce56afe-g94dz\" (UID: \"3066172f-0a14-486e-9cba-a2177d0760ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce56afe-g94dz"
Nov 02 01:55:36.715068 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:36.715028    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtswz\" (UniqueName: \"kubernetes.io/projected/3066172f-0a14-486e-9cba-a2177d0760ce-kube-api-access-jtswz\") pod \"maas-api-revoke-sub-fce56afe-g94dz\" (UID: \"3066172f-0a14-486e-9cba-a2177d0760ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce56afe-g94dz"
Nov 02 01:55:36.718719 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:36.717096    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3066172f-0a14-486e-9cba-a2177d0760ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fce56afe-g94dz\" (UID: \"3066172f-0a14-486e-9cba-a2177d0760ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce56afe-g94dz"
Nov 02 01:55:36.738368 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:36.736647    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3066172f-0a14-486e-9cba-a2177d0760ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fce56afe-g94dz\" (UID: \"3066172f-0a14-486e-9cba-a2177d0760ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce56afe-g94dz"
Nov 02 01:55:36.783458 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:36.783424    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtswz\" (UniqueName: \"kubernetes.io/projected/3066172f-0a14-486e-9cba-a2177d0760ce-kube-api-access-jtswz\") pod \"maas-api-revoke-sub-fce56afe-g94dz\" (UID: \"3066172f-0a14-486e-9cba-a2177d0760ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce56afe-g94dz"
Nov 02 01:55:36.800511 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:36.800483    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce56afe-g94dz"
Nov 02 01:55:37.193299 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:37.193253    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fce56afe-g94dz"]
Nov 02 01:55:37.194171 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:55:37.194141    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3066172f_0a14_486e_9cba_a2177d0760ce.slice/crio-b937db5d3902b6f442c6ae84347c0710e1756a5f5f31f2c3e4c2792b371b80f2 WatchSource:0}: Error finding container b937db5d3902b6f442c6ae84347c0710e1756a5f5f31f2c3e4c2792b371b80f2: Status 404 returned error can't find the container with id b937db5d3902b6f442c6ae84347c0710e1756a5f5f31f2c3e4c2792b371b80f2
Nov 02 01:55:37.221182 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:37.221152    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f975458f-bee6-46a6-a92c-189a9509d0e6" path="/var/lib/kubelet/pods/f975458f-bee6-46a6-a92c-189a9509d0e6/volumes"
Nov 02 01:55:37.320238 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:37.320206    5639 scope.go:122] "RemoveContainer" containerID="c5279e59befdebb76f7a6198bcb4efcc288f8728d5e7aca87757353792db2271"
Nov 02 01:55:37.330701 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:37.330677    5639 scope.go:122] "RemoveContainer" containerID="4635aa259a44fee93719d3edbe64ac42838f164d5d505073a29850fdc8f7ce1c"
Nov 02 01:55:37.340156 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:37.340134    5639 scope.go:122] "RemoveContainer" containerID="7759776f4a462e961d68eb80f6e7c1dc5731bf450e3ecdd3d093be4f414e7a3a"
Nov 02 01:55:37.351889 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:37.351870    5639 scope.go:122] "RemoveContainer" containerID="f24f14cc5087e85696f02ac7e95cc8a57a074142ca56b09c5bb7508f04be35c2"
Nov 02 01:55:37.437449 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:37.437416    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce56afe-g94dz" event={"ID":"3066172f-0a14-486e-9cba-a2177d0760ce","Type":"ContainerStarted","Data":"b937db5d3902b6f442c6ae84347c0710e1756a5f5f31f2c3e4c2792b371b80f2"}
Nov 02 01:55:38.448553 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:38.448513    5639 generic.go:356] "Generic (PLEG): container finished" podID="3066172f-0a14-486e-9cba-a2177d0760ce" containerID="885fe77a7882c680929f4d1381c8c1fc8b63a44a0d942b9733ec814a0711222d" exitCode=0
Nov 02 01:55:38.448969 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:38.448574    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce56afe-g94dz" event={"ID":"3066172f-0a14-486e-9cba-a2177d0760ce","Type":"ContainerDied","Data":"885fe77a7882c680929f4d1381c8c1fc8b63a44a0d942b9733ec814a0711222d"}
Nov 02 01:55:39.637649 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:39.637619    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce56afe-g94dz"
Nov 02 01:55:39.759450 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:39.759360    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3066172f-0a14-486e-9cba-a2177d0760ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3066172f-0a14-486e-9cba-a2177d0760ce-maas-api-xks-ca\") pod \"3066172f-0a14-486e-9cba-a2177d0760ce\" (UID: \"3066172f-0a14-486e-9cba-a2177d0760ce\") "
Nov 02 01:55:39.759628 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:39.759609    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3066172f-0a14-486e-9cba-a2177d0760ce-kube-api-access-jtswz\" (UniqueName: \"kubernetes.io/projected/3066172f-0a14-486e-9cba-a2177d0760ce-kube-api-access-jtswz\") pod \"3066172f-0a14-486e-9cba-a2177d0760ce\" (UID: \"3066172f-0a14-486e-9cba-a2177d0760ce\") "
Nov 02 01:55:39.759826 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:39.759808    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3066172f-0a14-486e-9cba-a2177d0760ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3066172f-0a14-486e-9cba-a2177d0760ce-maas-api-openshift-ca\") pod \"3066172f-0a14-486e-9cba-a2177d0760ce\" (UID: \"3066172f-0a14-486e-9cba-a2177d0760ce\") "
Nov 02 01:55:39.768897 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:39.765525    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3066172f-0a14-486e-9cba-a2177d0760ce-maas-api-openshift-ca" pod "3066172f-0a14-486e-9cba-a2177d0760ce" (UID: "3066172f-0a14-486e-9cba-a2177d0760ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:55:39.773470 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:39.771345    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3066172f-0a14-486e-9cba-a2177d0760ce-maas-api-xks-ca" pod "3066172f-0a14-486e-9cba-a2177d0760ce" (UID: "3066172f-0a14-486e-9cba-a2177d0760ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:55:39.774113 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:39.774071    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3066172f-0a14-486e-9cba-a2177d0760ce-kube-api-access-jtswz" pod "3066172f-0a14-486e-9cba-a2177d0760ce" (UID: "3066172f-0a14-486e-9cba-a2177d0760ce"). InnerVolumeSpecName "kube-api-access-jtswz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:55:39.862504 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:39.862464    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtswz\" (UniqueName: \"kubernetes.io/projected/3066172f-0a14-486e-9cba-a2177d0760ce-kube-api-access-jtswz\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:39.862504 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:39.862497    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3066172f-0a14-486e-9cba-a2177d0760ce-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:39.862729 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:39.862513    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3066172f-0a14-486e-9cba-a2177d0760ce-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:40.465440 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:40.465409    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce56afe-g94dz"
Nov 02 01:55:40.465626 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:40.465436    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce56afe-g94dz" event={"ID":"3066172f-0a14-486e-9cba-a2177d0760ce","Type":"ContainerDied","Data":"b937db5d3902b6f442c6ae84347c0710e1756a5f5f31f2c3e4c2792b371b80f2"}
Nov 02 01:55:40.465626 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:40.465486    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b937db5d3902b6f442c6ae84347c0710e1756a5f5f31f2c3e4c2792b371b80f2"
Nov 02 01:55:41.786679 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:41.786643    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:55:45.421845 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:45.421764    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r"
Nov 02 01:55:45.421845 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:45.421844    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf"
Nov 02 01:55:52.214453 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:52.214416    5639 patch_prober.go:29] interesting pod/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:52.214936 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:52.214475    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g" podUID="70900cc0-155e-46f3-bf31-14169d79b332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:53.651710 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:53.651679    5639 generic.go:356] "Generic (PLEG): container finished" podID="70900cc0-155e-46f3-bf31-14169d79b332" containerID="8964a3324e240077aca0c73deb70a63366b0077d5e482387449a2cedc0b79ca5" exitCode=0
Nov 02 01:55:53.652173 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:53.651757    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g" event={"ID":"70900cc0-155e-46f3-bf31-14169d79b332","Type":"ContainerDied","Data":"8964a3324e240077aca0c73deb70a63366b0077d5e482387449a2cedc0b79ca5"}
Nov 02 01:55:53.652225 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:53.652200    5639 scope.go:122] "RemoveContainer" containerID="8964a3324e240077aca0c73deb70a63366b0077d5e482387449a2cedc0b79ca5"
Nov 02 01:55:53.653866 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:53.653835    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:55:53.653940 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:53.653907    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:55:53.653976 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:53.653939    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:55:54.686871 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:54.686831    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g" event={"ID":"70900cc0-155e-46f3-bf31-14169d79b332","Type":"ContainerStarted","Data":"d9e37ccbf6a89c8ecc79d81a2498e58f3bdfceb69c2e50f17ddd79db6fcb0e5c"}
Nov 02 01:55:54.981374 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:54.980712    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:55:54.981374 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:54.980851    5639 patch_prober.go:29] interesting pod/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Nov 02 01:55:54.981374 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:54.980893    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g" podUID="70900cc0-155e-46f3-bf31-14169d79b332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Nov 02 01:55:55.984309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:55.984136    5639 patch_prober.go:29] interesting pod/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Nov 02 01:55:55.984309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:55.984215    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g" podUID="70900cc0-155e-46f3-bf31-14169d79b332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Nov 02 01:55:56.985304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:56.985183    5639 patch_prober.go:29] interesting pod/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Nov 02 01:55:56.985304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:56.985254    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g" podUID="70900cc0-155e-46f3-bf31-14169d79b332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Nov 02 01:55:57.422564 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:55:57.422527    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:55:57.984722 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:57.984672    5639 patch_prober.go:29] interesting pod/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Nov 02 01:55:57.984950 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:57.984749    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g" podUID="70900cc0-155e-46f3-bf31-14169d79b332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Nov 02 01:55:58.984319 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:58.983780    5639 patch_prober.go:29] interesting pod/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Nov 02 01:55:58.984319 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:58.983841    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g" podUID="70900cc0-155e-46f3-bf31-14169d79b332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Nov 02 01:55:59.998853 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:59.998817    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:55:59.999521 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:55:59.999357    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:56:00.000116 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:00.000097    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 01:56:03.017986 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.017949    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"]
Nov 02 01:56:03.018726 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.018709    5639 cpu_manager.go:414] "containerMap: removing container" podUID="3066172f-0a14-486e-9cba-a2177d0760ce" containerName="revoke-keys"
Nov 02 01:56:03.018776 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.018727    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3066172f-0a14-486e-9cba-a2177d0760ce" containerName="revoke-keys"
Nov 02 01:56:03.018869 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.018860    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="3066172f-0a14-486e-9cba-a2177d0760ce" containerName="revoke-keys"
Nov 02 01:56:03.022579 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.022561    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"
Nov 02 01:56:03.053661 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.053624    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-dshm\") pod \"e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr\" (UID: \"983fcbba-4b11-4119-a1da-9d340c30ca51\") " pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"
Nov 02 01:56:03.053839 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.053685    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/983fcbba-4b11-4119-a1da-9d340c30ca51-tls-certs\") pod \"e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr\" (UID: \"983fcbba-4b11-4119-a1da-9d340c30ca51\") " pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"
Nov 02 01:56:03.053839 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.053802    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkxgr\" (UniqueName: \"kubernetes.io/projected/983fcbba-4b11-4119-a1da-9d340c30ca51-kube-api-access-bkxgr\") pod \"e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr\" (UID: \"983fcbba-4b11-4119-a1da-9d340c30ca51\") " pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"
Nov 02 01:56:03.053938 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.053905    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-home\") pod \"e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr\" (UID: \"983fcbba-4b11-4119-a1da-9d340c30ca51\") " pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"
Nov 02 01:56:03.054004 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.053987    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-model-cache\") pod \"e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr\" (UID: \"983fcbba-4b11-4119-a1da-9d340c30ca51\") " pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"
Nov 02 01:56:03.054201 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.054177    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-tmp-dir\") pod \"e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr\" (UID: \"983fcbba-4b11-4119-a1da-9d340c30ca51\") " pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"
Nov 02 01:56:03.092401 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.092326    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-07ac70d0\"/\"kube-root-ca.crt\""
Nov 02 01:56:03.125072 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.125034    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-07ac70d0\"/\"openshift-service-ca.crt\""
Nov 02 01:56:03.125260 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.125090    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-07ac70d0\"/\"default-dockercfg-b26l9\""
Nov 02 01:56:03.125260 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.125136    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-07ac70d0\"/\"e2e-lifecycle-model-07ac70d0-kserve-self-signed-certs\""
Nov 02 01:56:03.129289 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.129254    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"]
Nov 02 01:56:03.166311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.157489    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-dshm\") pod \"e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr\" (UID: \"983fcbba-4b11-4119-a1da-9d340c30ca51\") " pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"
Nov 02 01:56:03.166311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.157545    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/983fcbba-4b11-4119-a1da-9d340c30ca51-tls-certs\") pod \"e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr\" (UID: \"983fcbba-4b11-4119-a1da-9d340c30ca51\") " pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"
Nov 02 01:56:03.166311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.157654    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkxgr\" (UniqueName: \"kubernetes.io/projected/983fcbba-4b11-4119-a1da-9d340c30ca51-kube-api-access-bkxgr\") pod \"e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr\" (UID: \"983fcbba-4b11-4119-a1da-9d340c30ca51\") " pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"
Nov 02 01:56:03.166311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.157749    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-home\") pod \"e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr\" (UID: \"983fcbba-4b11-4119-a1da-9d340c30ca51\") " pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"
Nov 02 01:56:03.166311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.157828    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-model-cache\") pod \"e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr\" (UID: \"983fcbba-4b11-4119-a1da-9d340c30ca51\") " pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"
Nov 02 01:56:03.166311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.157870    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-tmp-dir\") pod \"e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr\" (UID: \"983fcbba-4b11-4119-a1da-9d340c30ca51\") " pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"
Nov 02 01:56:03.166311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.158463    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-tmp-dir\") pod \"e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr\" (UID: \"983fcbba-4b11-4119-a1da-9d340c30ca51\") " pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"
Nov 02 01:56:03.170411 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.170377    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-dshm\") pod \"e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr\" (UID: \"983fcbba-4b11-4119-a1da-9d340c30ca51\") " pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"
Nov 02 01:56:03.170879 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.170854    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-home\") pod \"e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr\" (UID: \"983fcbba-4b11-4119-a1da-9d340c30ca51\") " pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"
Nov 02 01:56:03.171273 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.171253    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-model-cache\") pod \"e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr\" (UID: \"983fcbba-4b11-4119-a1da-9d340c30ca51\") " pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"
Nov 02 01:56:03.210454 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.201595    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/983fcbba-4b11-4119-a1da-9d340c30ca51-tls-certs\") pod \"e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr\" (UID: \"983fcbba-4b11-4119-a1da-9d340c30ca51\") " pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"
Nov 02 01:56:03.282358 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.270983    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkxgr\" (UniqueName: \"kubernetes.io/projected/983fcbba-4b11-4119-a1da-9d340c30ca51-kube-api-access-bkxgr\") pod \"e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr\" (UID: \"983fcbba-4b11-4119-a1da-9d340c30ca51\") " pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"
Nov 02 01:56:03.335415 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.335382    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"
Nov 02 01:56:03.734926 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.734874    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"]
Nov 02 01:56:03.753481 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:03.753451    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr" event={"ID":"983fcbba-4b11-4119-a1da-9d340c30ca51","Type":"ContainerStarted","Data":"0b3eee272d3e6fea8e7f2b541624958aac49e73f6ab2dbc776fec84b25b46fff"}
Nov 02 01:56:04.050234 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:04.050150    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31955345-qdbvw"]
Nov 02 01:56:04.053635 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:04.053611    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31955345-qdbvw"]
Nov 02 01:56:04.763119 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:04.763084    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr" event={"ID":"983fcbba-4b11-4119-a1da-9d340c30ca51","Type":"ContainerStarted","Data":"525917f823e3a88b45bf6aabde23b8005deee742b2a7d07f3bfb6f3ab5e32bae"}
Nov 02 01:56:04.881115 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:04.881050    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr" podStartSLOduration=2.881036342 podStartE2EDuration="2.881036342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:56:04.878756829 +0000 UTC m=+1829.782438943" watchObservedRunningTime="2026-11-02 01:56:04.881036342 +0000 UTC m=+1829.784718455"
Nov 02 01:56:05.220032 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:05.219997    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f24c71c-bafc-46fa-b9f4-6ee006c40953" path="/var/lib/kubelet/pods/3f24c71c-bafc-46fa-b9f4-6ee006c40953/volumes"
Nov 02 01:56:05.336205 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:05.336177    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"
Nov 02 01:56:08.004883 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:08.004845    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv"]
Nov 02 01:56:08.010581 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:08.010562    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv"
Nov 02 01:56:08.067489 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:08.067455    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv"]
Nov 02 01:56:08.111189 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:08.111157    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:56:08.111189 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:08.111157    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:56:08.111189 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:08.111186    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:56:08.112194 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:08.112161    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:56:08.163761 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:08.163724    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggpht\" (UniqueName: \"kubernetes.io/projected/c3b8159b-f16f-4a14-8077-255093cabef8-kube-api-access-ggpht\") pod \"maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv\" (UID: \"c3b8159b-f16f-4a14-8077-255093cabef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv"
Nov 02 01:56:08.163940 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:08.163773    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b8159b-f16f-4a14-8077-255093cabef8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv\" (UID: \"c3b8159b-f16f-4a14-8077-255093cabef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv"
Nov 02 01:56:08.163940 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:08.163900    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b8159b-f16f-4a14-8077-255093cabef8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv\" (UID: \"c3b8159b-f16f-4a14-8077-255093cabef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv"
Nov 02 01:56:08.265860 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:08.265776    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggpht\" (UniqueName: \"kubernetes.io/projected/c3b8159b-f16f-4a14-8077-255093cabef8-kube-api-access-ggpht\") pod \"maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv\" (UID: \"c3b8159b-f16f-4a14-8077-255093cabef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv"
Nov 02 01:56:08.265860 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:08.265816    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b8159b-f16f-4a14-8077-255093cabef8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv\" (UID: \"c3b8159b-f16f-4a14-8077-255093cabef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv"
Nov 02 01:56:08.266104 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:08.265878    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b8159b-f16f-4a14-8077-255093cabef8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv\" (UID: \"c3b8159b-f16f-4a14-8077-255093cabef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv"
Nov 02 01:56:08.266608 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:08.266583    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b8159b-f16f-4a14-8077-255093cabef8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv\" (UID: \"c3b8159b-f16f-4a14-8077-255093cabef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv"
Nov 02 01:56:08.268472 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:08.268454    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b8159b-f16f-4a14-8077-255093cabef8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv\" (UID: \"c3b8159b-f16f-4a14-8077-255093cabef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv"
Nov 02 01:56:08.328381 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:08.328347    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggpht\" (UniqueName: \"kubernetes.io/projected/c3b8159b-f16f-4a14-8077-255093cabef8-kube-api-access-ggpht\") pod \"maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv\" (UID: \"c3b8159b-f16f-4a14-8077-255093cabef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv"
Nov 02 01:56:08.625730 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:08.625696    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv"
Nov 02 01:56:09.007826 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:09.007793    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv"]
Nov 02 01:56:09.008248 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:56:09.008140    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b8159b_f16f_4a14_8077_255093cabef8.slice/crio-13c1e4257773971500fedc3c8c177105a8fa89e86d4b432054b1e6f36a70ee4b WatchSource:0}: Error finding container 13c1e4257773971500fedc3c8c177105a8fa89e86d4b432054b1e6f36a70ee4b: Status 404 returned error can't find the container with id 13c1e4257773971500fedc3c8c177105a8fa89e86d4b432054b1e6f36a70ee4b
Nov 02 01:56:09.908399 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:09.908362    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv" event={"ID":"c3b8159b-f16f-4a14-8077-255093cabef8","Type":"ContainerStarted","Data":"9f6fb0ca2b1ee9134dc34d5355bdcb81ea057d81ea41fa774ef54fff9fe06bbd"}
Nov 02 01:56:09.908588 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:09.908426    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv" event={"ID":"c3b8159b-f16f-4a14-8077-255093cabef8","Type":"ContainerStarted","Data":"13c1e4257773971500fedc3c8c177105a8fa89e86d4b432054b1e6f36a70ee4b"}
Nov 02 01:56:10.025086 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:10.025033    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv" podStartSLOduration=3.025018183 podStartE2EDuration="3.025018183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:56:10.022827294 +0000 UTC m=+1834.926509409" watchObservedRunningTime="2026-11-02 01:56:10.025018183 +0000 UTC m=+1834.928700334"
Nov 02 01:56:10.918269 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:10.918233    5639 generic.go:356] "Generic (PLEG): container finished" podID="c3b8159b-f16f-4a14-8077-255093cabef8" containerID="9f6fb0ca2b1ee9134dc34d5355bdcb81ea057d81ea41fa774ef54fff9fe06bbd" exitCode=0
Nov 02 01:56:10.918498 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:10.918340    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv" event={"ID":"c3b8159b-f16f-4a14-8077-255093cabef8","Type":"ContainerDied","Data":"9f6fb0ca2b1ee9134dc34d5355bdcb81ea057d81ea41fa774ef54fff9fe06bbd"}
Nov 02 01:56:11.984677 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:11.984643    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"]
Nov 02 01:56:11.989429 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:11.989403    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.133361 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.067159    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-token\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.214443 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.214355    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-data\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.214994 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.206856    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"]
Nov 02 01:56:12.215253 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.120638    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-39ceb2-openshift-default-dockercfg-fwdsf\""
Nov 02 01:56:12.215781 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.215762    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istiod-ca-cert\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.233498 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.233461    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-envoy\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.235445 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.235412    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nvxp\" (UniqueName: \"kubernetes.io/projected/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-kube-api-access-8nvxp\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.235700 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.235683    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-workload-certs\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.236016 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.235997    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-workload-socket\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.240303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.236221    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-podinfo\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.240652 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.240625    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-credential-socket\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.363328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.348433    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-data\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.363328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.348552    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istiod-ca-cert\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.363328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.348659    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-envoy\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.363328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.348842    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nvxp\" (UniqueName: \"kubernetes.io/projected/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-kube-api-access-8nvxp\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.363328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.348872    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-workload-certs\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.363328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.348915    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-workload-socket\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.363328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.348966    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-podinfo\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.363328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.349005    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-credential-socket\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.363328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.349051    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-token\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.363328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.349826    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-data\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.363328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.355734    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-credential-socket\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.363328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.356306    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istiod-ca-cert\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.400485 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.393400    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-workload-certs\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.400485 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.394124    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-workload-socket\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.463132 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.463098    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-envoy\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.466117 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.465976    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nvxp\" (UniqueName: \"kubernetes.io/projected/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-kube-api-access-8nvxp\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.466587 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.466554    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-podinfo\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.473630 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.473607    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-token\") pod \"e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") " pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.550125 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.550034    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv"
Nov 02 01:56:12.600309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.600169    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:12.680972 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.680488    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3b8159b-f16f-4a14-8077-255093cabef8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b8159b-f16f-4a14-8077-255093cabef8-maas-api-xks-ca\") pod \"c3b8159b-f16f-4a14-8077-255093cabef8\" (UID: \"c3b8159b-f16f-4a14-8077-255093cabef8\") "
Nov 02 01:56:12.680972 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.680617    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3b8159b-f16f-4a14-8077-255093cabef8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b8159b-f16f-4a14-8077-255093cabef8-maas-api-openshift-ca\") pod \"c3b8159b-f16f-4a14-8077-255093cabef8\" (UID: \"c3b8159b-f16f-4a14-8077-255093cabef8\") "
Nov 02 01:56:12.680972 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.680698    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3b8159b-f16f-4a14-8077-255093cabef8-kube-api-access-ggpht\" (UniqueName: \"kubernetes.io/projected/c3b8159b-f16f-4a14-8077-255093cabef8-kube-api-access-ggpht\") pod \"c3b8159b-f16f-4a14-8077-255093cabef8\" (UID: \"c3b8159b-f16f-4a14-8077-255093cabef8\") "
Nov 02 01:56:12.692816 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.691564    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b8159b-f16f-4a14-8077-255093cabef8-maas-api-openshift-ca" pod "c3b8159b-f16f-4a14-8077-255093cabef8" (UID: "c3b8159b-f16f-4a14-8077-255093cabef8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:56:12.695307 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.695254    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b8159b-f16f-4a14-8077-255093cabef8-kube-api-access-ggpht" pod "c3b8159b-f16f-4a14-8077-255093cabef8" (UID: "c3b8159b-f16f-4a14-8077-255093cabef8"). InnerVolumeSpecName "kube-api-access-ggpht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:56:12.695447 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.695414    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3b8159b-f16f-4a14-8077-255093cabef8-maas-api-xks-ca" pod "c3b8159b-f16f-4a14-8077-255093cabef8" (UID: "c3b8159b-f16f-4a14-8077-255093cabef8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:56:12.783337 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.783244    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b8159b-f16f-4a14-8077-255093cabef8-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:12.783337 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.783273    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b8159b-f16f-4a14-8077-255093cabef8-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:12.783337 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.783299    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggpht\" (UniqueName: \"kubernetes.io/projected/c3b8159b-f16f-4a14-8077-255093cabef8-kube-api-access-ggpht\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:12.942969 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.942937    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv"
Nov 02 01:56:12.942969 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.942946    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv" event={"ID":"c3b8159b-f16f-4a14-8077-255093cabef8","Type":"ContainerDied","Data":"13c1e4257773971500fedc3c8c177105a8fa89e86d4b432054b1e6f36a70ee4b"}
Nov 02 01:56:12.943235 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:12.943007    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13c1e4257773971500fedc3c8c177105a8fa89e86d4b432054b1e6f36a70ee4b"
Nov 02 01:56:13.055207 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:13.055174    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"]
Nov 02 01:56:13.058465 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:13.058430    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:56:13.058572 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:13.058511    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:56:13.058572 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:13.058558    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:56:13.336067 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:13.336042    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"
Nov 02 01:56:13.348102 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:13.348081    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"
Nov 02 01:56:13.830945 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:13.830901    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"]
Nov 02 01:56:13.831139 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:13.831117    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz" podUID="b4bfc70f-fa52-4876-9699-56040fd06e57" containerName="istio-proxy" containerID="cri-o://08b6aea114889adf04b7352833b29ef17ea0dc8d1b662b031adf295a951c9a5e" gracePeriod=30
Nov 02 01:56:13.834850 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:13.834825    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"]
Nov 02 01:56:13.835057 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:13.835029    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f" podUID="94b58fbf-2abd-4e6e-b40c-f7dcd8006713" containerName="istio-proxy" containerID="cri-o://eabb6763d3cfe188665b9843bda64fd0788154b7f7b0cbb5e7981705f310d023" gracePeriod=30
Nov 02 01:56:13.881154 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:13.881121    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"]
Nov 02 01:56:13.881398 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:13.881375    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2" podUID="c4dec6db-d72f-45ce-802c-f54bb7f57e5c" containerName="istio-proxy" containerID="cri-o://4e7d9c762d5d8ba4b4170e6817722b48b66635bad48a192409959689c9bc9469" gracePeriod=30
Nov 02 01:56:13.885220 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:13.885191    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"]
Nov 02 01:56:13.885588 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:13.885560    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw" podUID="cbdac3f5-20af-4282-b567-66a4d880c3b4" containerName="istio-proxy" containerID="cri-o://c8367e6fda706d5a07d98002b1fbd959bffe814a3f89b2ea5701ecc8669eff17" gracePeriod=30
Nov 02 01:56:13.961159 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:13.961005    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z" event={"ID":"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d","Type":"ContainerStarted","Data":"c7546833bfb3c3832e8221acfb033a02a8ff85aca3e16399304f8f04bad8afc0"}
Nov 02 01:56:13.961159 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:13.961095    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z" event={"ID":"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d","Type":"ContainerStarted","Data":"72d722099259df493de585a32403456907fde98f72748e24dd1166dd6263ccd6"}
Nov 02 01:56:13.973102 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:13.973074    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"
Nov 02 01:56:14.433461 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:14.433405    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z" podStartSLOduration=3.433386765 podStartE2EDuration="3.433386765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:56:14.431751539 +0000 UTC m=+1839.335433654" watchObservedRunningTime="2026-11-02 01:56:14.433386765 +0000 UTC m=+1839.337068879"
Nov 02 01:56:14.617323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:14.603824    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:14.630632 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:14.630599    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:14.969861 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:14.969828    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:14.970708 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:14.970687    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:56:16.391649 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:16.391614    5639 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:56:16.392048 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:16.391672    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2" podUID="c4dec6db-d72f-45ce-802c-f54bb7f57e5c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:56:19.014123 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.014077    5639 generic.go:356] "Generic (PLEG): container finished" podID="cbdac3f5-20af-4282-b567-66a4d880c3b4" containerID="c8367e6fda706d5a07d98002b1fbd959bffe814a3f89b2ea5701ecc8669eff17" exitCode=0
Nov 02 01:56:19.014559 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.014268    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw" event={"ID":"cbdac3f5-20af-4282-b567-66a4d880c3b4","Type":"ContainerDied","Data":"c8367e6fda706d5a07d98002b1fbd959bffe814a3f89b2ea5701ecc8669eff17"}
Nov 02 01:56:19.018885 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.018852    5639 generic.go:356] "Generic (PLEG): container finished" podID="94b58fbf-2abd-4e6e-b40c-f7dcd8006713" containerID="eabb6763d3cfe188665b9843bda64fd0788154b7f7b0cbb5e7981705f310d023" exitCode=0
Nov 02 01:56:19.019120 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.018972    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f" event={"ID":"94b58fbf-2abd-4e6e-b40c-f7dcd8006713","Type":"ContainerDied","Data":"eabb6763d3cfe188665b9843bda64fd0788154b7f7b0cbb5e7981705f310d023"}
Nov 02 01:56:19.034241 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.027247    5639 generic.go:356] "Generic (PLEG): container finished" podID="c4dec6db-d72f-45ce-802c-f54bb7f57e5c" containerID="4e7d9c762d5d8ba4b4170e6817722b48b66635bad48a192409959689c9bc9469" exitCode=0
Nov 02 01:56:19.034241 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.027537    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2" event={"ID":"c4dec6db-d72f-45ce-802c-f54bb7f57e5c","Type":"ContainerDied","Data":"4e7d9c762d5d8ba4b4170e6817722b48b66635bad48a192409959689c9bc9469"}
Nov 02 01:56:19.034241 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.027654    5639 scope.go:122] "RemoveContainer" containerID="686c26bc415f61deefbc110374f8064e90e1eb57966696875b12338d48775fbd"
Nov 02 01:56:19.034241 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.032141    5639 generic.go:356] "Generic (PLEG): container finished" podID="b4bfc70f-fa52-4876-9699-56040fd06e57" containerID="08b6aea114889adf04b7352833b29ef17ea0dc8d1b662b031adf295a951c9a5e" exitCode=0
Nov 02 01:56:19.034241 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.032222    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz" event={"ID":"b4bfc70f-fa52-4876-9699-56040fd06e57","Type":"ContainerDied","Data":"08b6aea114889adf04b7352833b29ef17ea0dc8d1b662b031adf295a951c9a5e"}
Nov 02 01:56:19.446438 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.446033    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:56:19.495304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.495245    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n"]
Nov 02 01:56:19.574398 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.574362    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:56:19.579322 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.579091    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:56:19.583672 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.583647    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-podinfo\") pod \"b4bfc70f-fa52-4876-9699-56040fd06e57\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") "
Nov 02 01:56:19.583806 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.583686    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-envoy\") pod \"b4bfc70f-fa52-4876-9699-56040fd06e57\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") "
Nov 02 01:56:19.583806 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.583717    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-token\" (UniqueName: \"kubernetes.io/projected/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-token\") pod \"b4bfc70f-fa52-4876-9699-56040fd06e57\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") "
Nov 02 01:56:19.583806 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.583795    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-data\") pod \"b4bfc70f-fa52-4876-9699-56040fd06e57\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") "
Nov 02 01:56:19.583973 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.583833    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4bfc70f-fa52-4876-9699-56040fd06e57-kube-api-access-94mkq\" (UniqueName: \"kubernetes.io/projected/b4bfc70f-fa52-4876-9699-56040fd06e57-kube-api-access-94mkq\") pod \"b4bfc70f-fa52-4876-9699-56040fd06e57\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") "
Nov 02 01:56:19.583973 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.583878    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-credential-socket\") pod \"b4bfc70f-fa52-4876-9699-56040fd06e57\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") "
Nov 02 01:56:19.584162 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.583973    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-workload-certs\") pod \"b4bfc70f-fa52-4876-9699-56040fd06e57\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") "
Nov 02 01:56:19.584162 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.584117    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-data" pod "b4bfc70f-fa52-4876-9699-56040fd06e57" (UID: "b4bfc70f-fa52-4876-9699-56040fd06e57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:19.584266 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.584239    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-workload-socket\") pod \"b4bfc70f-fa52-4876-9699-56040fd06e57\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") "
Nov 02 01:56:19.584370 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.584319    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4bfc70f-fa52-4876-9699-56040fd06e57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4bfc70f-fa52-4876-9699-56040fd06e57-istiod-ca-cert\") pod \"b4bfc70f-fa52-4876-9699-56040fd06e57\" (UID: \"b4bfc70f-fa52-4876-9699-56040fd06e57\") "
Nov 02 01:56:19.584717 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.584247    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:56:19.584943 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.584925    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.585536 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.585448    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-workload-certs" pod "b4bfc70f-fa52-4876-9699-56040fd06e57" (UID: "b4bfc70f-fa52-4876-9699-56040fd06e57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:19.585536 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.585469    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-credential-socket" pod "b4bfc70f-fa52-4876-9699-56040fd06e57" (UID: "b4bfc70f-fa52-4876-9699-56040fd06e57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:19.585694 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.585540    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-workload-socket" pod "b4bfc70f-fa52-4876-9699-56040fd06e57" (UID: "b4bfc70f-fa52-4876-9699-56040fd06e57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:19.585868 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.585845    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4bfc70f-fa52-4876-9699-56040fd06e57-istiod-ca-cert" pod "b4bfc70f-fa52-4876-9699-56040fd06e57" (UID: "b4bfc70f-fa52-4876-9699-56040fd06e57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:56:19.588358 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.588333    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-envoy" pod "b4bfc70f-fa52-4876-9699-56040fd06e57" (UID: "b4bfc70f-fa52-4876-9699-56040fd06e57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:19.589716 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.589691    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-podinfo" pod "b4bfc70f-fa52-4876-9699-56040fd06e57" (UID: "b4bfc70f-fa52-4876-9699-56040fd06e57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:56:19.590350 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.590323    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4bfc70f-fa52-4876-9699-56040fd06e57-kube-api-access-94mkq" pod "b4bfc70f-fa52-4876-9699-56040fd06e57" (UID: "b4bfc70f-fa52-4876-9699-56040fd06e57"). InnerVolumeSpecName "kube-api-access-94mkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:56:19.591549 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.591524    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-token" pod "b4bfc70f-fa52-4876-9699-56040fd06e57" (UID: "b4bfc70f-fa52-4876-9699-56040fd06e57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:56:19.631167 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.630568    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh"]
Nov 02 01:56:19.685624 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.685576    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istiod-ca-cert\") pod \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") "
Nov 02 01:56:19.685834 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.685630    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-workload-socket\") pod \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") "
Nov 02 01:56:19.685834 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.685672    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-workload-socket\") pod \"cbdac3f5-20af-4282-b567-66a4d880c3b4\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") "
Nov 02 01:56:19.685834 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.685739    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-kube-api-access-97tj9\" (UniqueName: \"kubernetes.io/projected/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-kube-api-access-97tj9\") pod \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") "
Nov 02 01:56:19.685834 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.685769    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-podinfo\") pod \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") "
Nov 02 01:56:19.686077 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.685860    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-workload-socket\") pod \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") "
Nov 02 01:56:19.686077 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.685891    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-token\" (UniqueName: \"kubernetes.io/projected/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-token\") pod \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") "
Nov 02 01:56:19.686077 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.685932    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbdac3f5-20af-4282-b567-66a4d880c3b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbdac3f5-20af-4282-b567-66a4d880c3b4-istiod-ca-cert\") pod \"cbdac3f5-20af-4282-b567-66a4d880c3b4\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") "
Nov 02 01:56:19.686077 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.685976    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-workload-certs\") pod \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") "
Nov 02 01:56:19.686077 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.685983    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-workload-socket" pod "c4dec6db-d72f-45ce-802c-f54bb7f57e5c" (UID: "c4dec6db-d72f-45ce-802c-f54bb7f57e5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:19.686077 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.686004    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-credential-socket\") pod \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") "
Nov 02 01:56:19.686077 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.686038    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-envoy\") pod \"cbdac3f5-20af-4282-b567-66a4d880c3b4\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") "
Nov 02 01:56:19.686518 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.686098    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-envoy\") pod \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") "
Nov 02 01:56:19.686518 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.686130    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-workload-certs\") pod \"cbdac3f5-20af-4282-b567-66a4d880c3b4\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") "
Nov 02 01:56:19.686518 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.686175    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-credential-socket\") pod \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") "
Nov 02 01:56:19.686518 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.686209    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-data\") pod \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") "
Nov 02 01:56:19.686518 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.686272    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-data\") pod \"cbdac3f5-20af-4282-b567-66a4d880c3b4\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") "
Nov 02 01:56:19.686518 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.686328    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-token\" (UniqueName: \"kubernetes.io/projected/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-token\") pod \"cbdac3f5-20af-4282-b567-66a4d880c3b4\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") "
Nov 02 01:56:19.686518 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.686368    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-data\") pod \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") "
Nov 02 01:56:19.686518 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.686427    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-token\" (UniqueName: \"kubernetes.io/projected/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-token\") pod \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") "
Nov 02 01:56:19.686938 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.686517    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-podinfo\") pod \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") "
Nov 02 01:56:19.686938 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.686565    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-kube-api-access-t67jl\" (UniqueName: \"kubernetes.io/projected/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-kube-api-access-t67jl\") pod \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") "
Nov 02 01:56:19.686938 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.686620    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-envoy\") pod \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") "
Nov 02 01:56:19.686938 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.686653    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-workload-certs\") pod \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\" (UID: \"c4dec6db-d72f-45ce-802c-f54bb7f57e5c\") "
Nov 02 01:56:19.686938 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.686763    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istiod-ca-cert\") pod \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\" (UID: \"94b58fbf-2abd-4e6e-b40c-f7dcd8006713\") "
Nov 02 01:56:19.686938 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.686844    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-podinfo\") pod \"cbdac3f5-20af-4282-b567-66a4d880c3b4\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") "
Nov 02 01:56:19.686938 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.686879    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbdac3f5-20af-4282-b567-66a4d880c3b4-kube-api-access-s2hpp\" (UniqueName: \"kubernetes.io/projected/cbdac3f5-20af-4282-b567-66a4d880c3b4-kube-api-access-s2hpp\") pod \"cbdac3f5-20af-4282-b567-66a4d880c3b4\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") "
Nov 02 01:56:19.686938 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.686928    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-credential-socket\") pod \"cbdac3f5-20af-4282-b567-66a4d880c3b4\" (UID: \"cbdac3f5-20af-4282-b567-66a4d880c3b4\") "
Nov 02 01:56:19.691759 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.687869    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.691759 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.687894    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.691759 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.687909    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4bfc70f-fa52-4876-9699-56040fd06e57-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.691759 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.687923    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.691759 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.687938    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.691759 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.687952    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.691759 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.687966    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4bfc70f-fa52-4876-9699-56040fd06e57-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.691759 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.687980    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94mkq\" (UniqueName: \"kubernetes.io/projected/b4bfc70f-fa52-4876-9699-56040fd06e57-kube-api-access-94mkq\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.691759 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.687995    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc70f-fa52-4876-9699-56040fd06e57-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.691759 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.686031    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istiod-ca-cert" pod "c4dec6db-d72f-45ce-802c-f54bb7f57e5c" (UID: "c4dec6db-d72f-45ce-802c-f54bb7f57e5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:56:19.691759 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.686205    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-workload-socket" pod "94b58fbf-2abd-4e6e-b40c-f7dcd8006713" (UID: "94b58fbf-2abd-4e6e-b40c-f7dcd8006713"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:19.691759 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.686448    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-workload-socket" pod "cbdac3f5-20af-4282-b567-66a4d880c3b4" (UID: "cbdac3f5-20af-4282-b567-66a4d880c3b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:19.691759 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.686490    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbdac3f5-20af-4282-b567-66a4d880c3b4-istiod-ca-cert" pod "cbdac3f5-20af-4282-b567-66a4d880c3b4" (UID: "cbdac3f5-20af-4282-b567-66a4d880c3b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:56:19.691759 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.687274    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-credential-socket" pod "cbdac3f5-20af-4282-b567-66a4d880c3b4" (UID: "cbdac3f5-20af-4282-b567-66a4d880c3b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:19.691759 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.688737    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-workload-certs" pod "c4dec6db-d72f-45ce-802c-f54bb7f57e5c" (UID: "c4dec6db-d72f-45ce-802c-f54bb7f57e5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:19.691759 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.689745    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-workload-certs" pod "cbdac3f5-20af-4282-b567-66a4d880c3b4" (UID: "cbdac3f5-20af-4282-b567-66a4d880c3b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:19.691759 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.689923    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-credential-socket" pod "94b58fbf-2abd-4e6e-b40c-f7dcd8006713" (UID: "94b58fbf-2abd-4e6e-b40c-f7dcd8006713"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:19.691759 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.689949    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-credential-socket" pod "c4dec6db-d72f-45ce-802c-f54bb7f57e5c" (UID: "c4dec6db-d72f-45ce-802c-f54bb7f57e5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:19.691759 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.690148    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-data" pod "c4dec6db-d72f-45ce-802c-f54bb7f57e5c" (UID: "c4dec6db-d72f-45ce-802c-f54bb7f57e5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:19.691759 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.690661    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-data" pod "cbdac3f5-20af-4282-b567-66a4d880c3b4" (UID: "cbdac3f5-20af-4282-b567-66a4d880c3b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:19.691759 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.691539    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-data" pod "94b58fbf-2abd-4e6e-b40c-f7dcd8006713" (UID: "94b58fbf-2abd-4e6e-b40c-f7dcd8006713"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:19.692964 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.691902    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-kube-api-access-97tj9" pod "94b58fbf-2abd-4e6e-b40c-f7dcd8006713" (UID: "94b58fbf-2abd-4e6e-b40c-f7dcd8006713"). InnerVolumeSpecName "kube-api-access-97tj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:56:19.692964 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.692583    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istiod-ca-cert" pod "94b58fbf-2abd-4e6e-b40c-f7dcd8006713" (UID: "94b58fbf-2abd-4e6e-b40c-f7dcd8006713"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:56:19.693067 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.693036    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-token" pod "94b58fbf-2abd-4e6e-b40c-f7dcd8006713" (UID: "94b58fbf-2abd-4e6e-b40c-f7dcd8006713"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:56:19.693122 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.693088    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-workload-certs" pod "94b58fbf-2abd-4e6e-b40c-f7dcd8006713" (UID: "94b58fbf-2abd-4e6e-b40c-f7dcd8006713"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:19.695710 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.695401    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-token" pod "cbdac3f5-20af-4282-b567-66a4d880c3b4" (UID: "cbdac3f5-20af-4282-b567-66a4d880c3b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:56:19.696016 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.695972    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-envoy" pod "cbdac3f5-20af-4282-b567-66a4d880c3b4" (UID: "cbdac3f5-20af-4282-b567-66a4d880c3b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:19.696328 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.696193    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-envoy" pod "94b58fbf-2abd-4e6e-b40c-f7dcd8006713" (UID: "94b58fbf-2abd-4e6e-b40c-f7dcd8006713"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:19.697349 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.697308    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-envoy" pod "c4dec6db-d72f-45ce-802c-f54bb7f57e5c" (UID: "c4dec6db-d72f-45ce-802c-f54bb7f57e5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:19.697726 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.697700    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-kube-api-access-t67jl" pod "c4dec6db-d72f-45ce-802c-f54bb7f57e5c" (UID: "c4dec6db-d72f-45ce-802c-f54bb7f57e5c"). InnerVolumeSpecName "kube-api-access-t67jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:56:19.697993 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.697974    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-podinfo" pod "c4dec6db-d72f-45ce-802c-f54bb7f57e5c" (UID: "c4dec6db-d72f-45ce-802c-f54bb7f57e5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:56:19.698218 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.698187    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbdac3f5-20af-4282-b567-66a4d880c3b4-kube-api-access-s2hpp" pod "cbdac3f5-20af-4282-b567-66a4d880c3b4" (UID: "cbdac3f5-20af-4282-b567-66a4d880c3b4"). InnerVolumeSpecName "kube-api-access-s2hpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:56:19.698941 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.698919    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-podinfo" pod "94b58fbf-2abd-4e6e-b40c-f7dcd8006713" (UID: "94b58fbf-2abd-4e6e-b40c-f7dcd8006713"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:56:19.698941 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.698927    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-token" pod "c4dec6db-d72f-45ce-802c-f54bb7f57e5c" (UID: "c4dec6db-d72f-45ce-802c-f54bb7f57e5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:56:19.699126 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.699103    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-podinfo" pod "cbdac3f5-20af-4282-b567-66a4d880c3b4" (UID: "cbdac3f5-20af-4282-b567-66a4d880c3b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:56:19.789196 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.789163    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.789196 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.789193    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.789196 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.789203    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.789478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.789212    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.789478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.789221    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.789478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.789230    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.789478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.789239    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.789478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.789249    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t67jl\" (UniqueName: \"kubernetes.io/projected/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-kube-api-access-t67jl\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.789478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.789259    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.789478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.789269    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.789478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.789297    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.789478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.789310    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.789478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.789322    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2hpp\" (UniqueName: \"kubernetes.io/projected/cbdac3f5-20af-4282-b567-66a4d880c3b4-kube-api-access-s2hpp\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.789478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.789336    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.789478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.789344    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.789478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.789352    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.789478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.789361    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97tj9\" (UniqueName: \"kubernetes.io/projected/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-kube-api-access-97tj9\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.789478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.789369    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.789478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.789377    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.789478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.789387    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.789478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.789397    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbdac3f5-20af-4282-b567-66a4d880c3b4-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.789478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.789406    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.789478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.789419    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b58fbf-2abd-4e6e-b40c-f7dcd8006713-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.789478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.789430    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.789478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.789439    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4dec6db-d72f-45ce-802c-f54bb7f57e5c-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.789478 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:19.789448    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbdac3f5-20af-4282-b567-66a4d880c3b4-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:20.072676 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:20.072585    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"
Nov 02 01:56:20.073383 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:20.073232    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw" event={"ID":"cbdac3f5-20af-4282-b567-66a4d880c3b4","Type":"ContainerDied","Data":"b9205aafeb821db85a3abcd35a180b7046c2e6d5f51ef35327bba43bda488126"}
Nov 02 01:56:20.073383 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:20.073357    5639 scope.go:122] "RemoveContainer" containerID="c8367e6fda706d5a07d98002b1fbd959bffe814a3f89b2ea5701ecc8669eff17"
Nov 02 01:56:20.076245 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:20.076225    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"
Nov 02 01:56:20.076769 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:20.076307    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f" event={"ID":"94b58fbf-2abd-4e6e-b40c-f7dcd8006713","Type":"ContainerDied","Data":"b56480489d81ce319b12c9233526ed737d7036183fd0ab683ec6d38e5c1e54e9"}
Nov 02 01:56:20.084591 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:20.078432    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"
Nov 02 01:56:20.084591 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:20.078433    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2" event={"ID":"c4dec6db-d72f-45ce-802c-f54bb7f57e5c","Type":"ContainerDied","Data":"2764475bdd37450dc8b3639d6f5c79d579b6a2835826cb63dd615473eabe96f7"}
Nov 02 01:56:20.086445 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:20.086034    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz" event={"ID":"b4bfc70f-fa52-4876-9699-56040fd06e57","Type":"ContainerDied","Data":"5327dfa7f0df0a92227c69830f2483e6f54bb2c3f7edee47d46d4f7ac1cbeffc"}
Nov 02 01:56:20.086445 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:20.086121    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"
Nov 02 01:56:20.089357 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:20.089248    5639 scope.go:122] "RemoveContainer" containerID="eabb6763d3cfe188665b9843bda64fd0788154b7f7b0cbb5e7981705f310d023"
Nov 02 01:56:20.127030 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:20.127006    5639 scope.go:122] "RemoveContainer" containerID="4e7d9c762d5d8ba4b4170e6817722b48b66635bad48a192409959689c9bc9469"
Nov 02 01:56:20.267418 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:20.267386    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"]
Nov 02 01:56:20.274670 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:20.272941    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-626p2"]
Nov 02 01:56:20.274670 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:20.274024    5639 scope.go:122] "RemoveContainer" containerID="08b6aea114889adf04b7352833b29ef17ea0dc8d1b662b031adf295a951c9a5e"
Nov 02 01:56:20.375082 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:20.375036    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"]
Nov 02 01:56:20.379659 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:20.379634    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vp8gw"]
Nov 02 01:56:20.657254 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:20.657176    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"]
Nov 02 01:56:20.663855 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:20.663822    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-gzq8f"]
Nov 02 01:56:20.774715 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:20.774673    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"]
Nov 02 01:56:20.778722 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:20.778688    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sx9rz"]
Nov 02 01:56:21.290797 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:21.290758    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94b58fbf-2abd-4e6e-b40c-f7dcd8006713" path="/var/lib/kubelet/pods/94b58fbf-2abd-4e6e-b40c-f7dcd8006713/volumes"
Nov 02 01:56:21.292345 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:21.292318    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4bfc70f-fa52-4876-9699-56040fd06e57" path="/var/lib/kubelet/pods/b4bfc70f-fa52-4876-9699-56040fd06e57/volumes"
Nov 02 01:56:21.294965 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:21.294941    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4dec6db-d72f-45ce-802c-f54bb7f57e5c" path="/var/lib/kubelet/pods/c4dec6db-d72f-45ce-802c-f54bb7f57e5c/volumes"
Nov 02 01:56:21.304343 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:21.304315    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbdac3f5-20af-4282-b567-66a4d880c3b4" path="/var/lib/kubelet/pods/cbdac3f5-20af-4282-b567-66a4d880c3b4/volumes"
Nov 02 01:56:24.497383 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:24.497313    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n" podUID="b65549a1-6aa9-46a7-a2e7-1407ff736f44" containerName="payload-processing" containerID="cri-o://e13e4fec26d258e921718bf15e0d0242162f30e9e287c3799ca0607aab0c164c" gracePeriod=25
Nov 02 01:56:24.632013 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:24.631971    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh" podUID="eba17c91-872f-468d-b14d-422f3fb73b4f" containerName="payload-pre-processing" containerID="cri-o://79fd48e08e9c9067d982849afa6de82b434d56ed08c58e349ef4b8ce95d5b740" gracePeriod=25
Nov 02 01:56:24.789718 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:24.789677    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n"
Nov 02 01:56:24.898767 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:24.898736    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b65549a1-6aa9-46a7-a2e7-1407ff736f44-kube-api-access-kxwsj\" (UniqueName: \"kubernetes.io/projected/b65549a1-6aa9-46a7-a2e7-1407ff736f44-kube-api-access-kxwsj\") pod \"b65549a1-6aa9-46a7-a2e7-1407ff736f44\" (UID: \"b65549a1-6aa9-46a7-a2e7-1407ff736f44\") "
Nov 02 01:56:24.898954 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:24.898938    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b65549a1-6aa9-46a7-a2e7-1407ff736f44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65549a1-6aa9-46a7-a2e7-1407ff736f44-plugins-config-volume\") pod \"b65549a1-6aa9-46a7-a2e7-1407ff736f44\" (UID: \"b65549a1-6aa9-46a7-a2e7-1407ff736f44\") "
Nov 02 01:56:24.899332 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:24.899274    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b65549a1-6aa9-46a7-a2e7-1407ff736f44-plugins-config-volume" pod "b65549a1-6aa9-46a7-a2e7-1407ff736f44" (UID: "b65549a1-6aa9-46a7-a2e7-1407ff736f44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:56:24.899482 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:24.899462    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65549a1-6aa9-46a7-a2e7-1407ff736f44-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:24.901051 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:24.901026    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65549a1-6aa9-46a7-a2e7-1407ff736f44-kube-api-access-kxwsj" pod "b65549a1-6aa9-46a7-a2e7-1407ff736f44" (UID: "b65549a1-6aa9-46a7-a2e7-1407ff736f44"). InnerVolumeSpecName "kube-api-access-kxwsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:56:24.920400 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:24.920372    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh"
Nov 02 01:56:25.000594 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:25.000561    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eba17c91-872f-468d-b14d-422f3fb73b4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eba17c91-872f-468d-b14d-422f3fb73b4f-plugins-config-volume\") pod \"eba17c91-872f-468d-b14d-422f3fb73b4f\" (UID: \"eba17c91-872f-468d-b14d-422f3fb73b4f\") "
Nov 02 01:56:25.000750 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:25.000664    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eba17c91-872f-468d-b14d-422f3fb73b4f-kube-api-access-5m59h\" (UniqueName: \"kubernetes.io/projected/eba17c91-872f-468d-b14d-422f3fb73b4f-kube-api-access-5m59h\") pod \"eba17c91-872f-468d-b14d-422f3fb73b4f\" (UID: \"eba17c91-872f-468d-b14d-422f3fb73b4f\") "
Nov 02 01:56:25.001052 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:25.001027    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eba17c91-872f-468d-b14d-422f3fb73b4f-plugins-config-volume" pod "eba17c91-872f-468d-b14d-422f3fb73b4f" (UID: "eba17c91-872f-468d-b14d-422f3fb73b4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:56:25.001097 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:25.001035    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxwsj\" (UniqueName: \"kubernetes.io/projected/b65549a1-6aa9-46a7-a2e7-1407ff736f44-kube-api-access-kxwsj\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:25.002988 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:25.002939    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba17c91-872f-468d-b14d-422f3fb73b4f-kube-api-access-5m59h" pod "eba17c91-872f-468d-b14d-422f3fb73b4f" (UID: "eba17c91-872f-468d-b14d-422f3fb73b4f"). InnerVolumeSpecName "kube-api-access-5m59h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:56:25.102363 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:25.102332    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m59h\" (UniqueName: \"kubernetes.io/projected/eba17c91-872f-468d-b14d-422f3fb73b4f-kube-api-access-5m59h\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:25.102363 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:25.102359    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eba17c91-872f-468d-b14d-422f3fb73b4f-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:25.303862 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:25.303777    5639 generic.go:356] "Generic (PLEG): container finished" podID="eba17c91-872f-468d-b14d-422f3fb73b4f" containerID="79fd48e08e9c9067d982849afa6de82b434d56ed08c58e349ef4b8ce95d5b740" exitCode=0
Nov 02 01:56:25.303862 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:25.303848    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh"
Nov 02 01:56:25.304069 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:25.303964    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh" event={"ID":"eba17c91-872f-468d-b14d-422f3fb73b4f","Type":"ContainerDied","Data":"79fd48e08e9c9067d982849afa6de82b434d56ed08c58e349ef4b8ce95d5b740"}
Nov 02 01:56:25.304069 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:25.304014    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh" event={"ID":"eba17c91-872f-468d-b14d-422f3fb73b4f","Type":"ContainerDied","Data":"a2756bbb63a5d448b9194c44dc88c476e9e792471c1ddd09a1065d7494d87c50"}
Nov 02 01:56:25.304069 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:25.304042    5639 scope.go:122] "RemoveContainer" containerID="79fd48e08e9c9067d982849afa6de82b434d56ed08c58e349ef4b8ce95d5b740"
Nov 02 01:56:25.306190 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:25.306164    5639 generic.go:356] "Generic (PLEG): container finished" podID="b65549a1-6aa9-46a7-a2e7-1407ff736f44" containerID="e13e4fec26d258e921718bf15e0d0242162f30e9e287c3799ca0607aab0c164c" exitCode=0
Nov 02 01:56:25.306388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:25.306210    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n"
Nov 02 01:56:25.306388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:25.306233    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n" event={"ID":"b65549a1-6aa9-46a7-a2e7-1407ff736f44","Type":"ContainerDied","Data":"e13e4fec26d258e921718bf15e0d0242162f30e9e287c3799ca0607aab0c164c"}
Nov 02 01:56:25.306388 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:25.306301    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n" event={"ID":"b65549a1-6aa9-46a7-a2e7-1407ff736f44","Type":"ContainerDied","Data":"5aa557112404d04506868591127e062d3c157cf30e87eb89b254d74fbcaa11c9"}
Nov 02 01:56:25.316234 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:25.316210    5639 scope.go:122] "RemoveContainer" containerID="79fd48e08e9c9067d982849afa6de82b434d56ed08c58e349ef4b8ce95d5b740"
Nov 02 01:56:25.316525 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:56:25.316508    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79fd48e08e9c9067d982849afa6de82b434d56ed08c58e349ef4b8ce95d5b740\": container with ID starting with 79fd48e08e9c9067d982849afa6de82b434d56ed08c58e349ef4b8ce95d5b740 not found: ID does not exist" containerID="79fd48e08e9c9067d982849afa6de82b434d56ed08c58e349ef4b8ce95d5b740"
Nov 02 01:56:25.316581 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:25.316534    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79fd48e08e9c9067d982849afa6de82b434d56ed08c58e349ef4b8ce95d5b740"} err="failed to get container status \"79fd48e08e9c9067d982849afa6de82b434d56ed08c58e349ef4b8ce95d5b740\": rpc error: code = NotFound desc = could not find container \"79fd48e08e9c9067d982849afa6de82b434d56ed08c58e349ef4b8ce95d5b740\": container with ID starting with 79fd48e08e9c9067d982849afa6de82b434d56ed08c58e349ef4b8ce95d5b740 not found: ID does not exist"
Nov 02 01:56:25.316581 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:25.316553    5639 scope.go:122] "RemoveContainer" containerID="e13e4fec26d258e921718bf15e0d0242162f30e9e287c3799ca0607aab0c164c"
Nov 02 01:56:25.326126 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:25.326104    5639 scope.go:122] "RemoveContainer" containerID="e13e4fec26d258e921718bf15e0d0242162f30e9e287c3799ca0607aab0c164c"
Nov 02 01:56:25.326450 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:56:25.326426    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e13e4fec26d258e921718bf15e0d0242162f30e9e287c3799ca0607aab0c164c\": container with ID starting with e13e4fec26d258e921718bf15e0d0242162f30e9e287c3799ca0607aab0c164c not found: ID does not exist" containerID="e13e4fec26d258e921718bf15e0d0242162f30e9e287c3799ca0607aab0c164c"
Nov 02 01:56:25.326517 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:25.326456    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e13e4fec26d258e921718bf15e0d0242162f30e9e287c3799ca0607aab0c164c"} err="failed to get container status \"e13e4fec26d258e921718bf15e0d0242162f30e9e287c3799ca0607aab0c164c\": rpc error: code = NotFound desc = could not find container \"e13e4fec26d258e921718bf15e0d0242162f30e9e287c3799ca0607aab0c164c\": container with ID starting with e13e4fec26d258e921718bf15e0d0242162f30e9e287c3799ca0607aab0c164c not found: ID does not exist"
Nov 02 01:56:25.473154 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:25.473110    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh"]
Nov 02 01:56:25.521879 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:25.521854    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-59842dd7-b7fbff559-5q9nh"]
Nov 02 01:56:25.587566 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:25.587534    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n"]
Nov 02 01:56:25.595573 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:25.595542    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-59842dd7-c5d9484c7-8jn8n"]
Nov 02 01:56:27.220609 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:27.220580    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b65549a1-6aa9-46a7-a2e7-1407ff736f44" path="/var/lib/kubelet/pods/b65549a1-6aa9-46a7-a2e7-1407ff736f44/volumes"
Nov 02 01:56:27.221052 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:27.221038    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eba17c91-872f-468d-b14d-422f3fb73b4f" path="/var/lib/kubelet/pods/eba17c91-872f-468d-b14d-422f3fb73b4f/volumes"
Nov 02 01:56:27.421488 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:56:27.421452    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:56:37.429063 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:37.428164    5639 scope.go:122] "RemoveContainer" containerID="c22524ed377e3dc969e68ef431eb38e511f4ecc12248fb3a5966127a0797b0b1"
Nov 02 01:56:37.491619 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:37.491481    5639 scope.go:122] "RemoveContainer" containerID="5a60374ed76a0dc1ce621130abf9562507f94e2d2c12312830b1a5be6ee97461"
Nov 02 01:56:40.787120 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.786585    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs"]
Nov 02 01:56:40.790583 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.790557    5639 cpu_manager.go:414] "containerMap: removing container" podUID="b4bfc70f-fa52-4876-9699-56040fd06e57" containerName="istio-proxy"
Nov 02 01:56:40.790583 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.790585    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4bfc70f-fa52-4876-9699-56040fd06e57" containerName="istio-proxy"
Nov 02 01:56:40.790765 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.790611    5639 cpu_manager.go:414] "containerMap: removing container" podUID="c4dec6db-d72f-45ce-802c-f54bb7f57e5c" containerName="istio-proxy"
Nov 02 01:56:40.790765 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.790620    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4dec6db-d72f-45ce-802c-f54bb7f57e5c" containerName="istio-proxy"
Nov 02 01:56:40.790765 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.790651    5639 cpu_manager.go:414] "containerMap: removing container" podUID="cbdac3f5-20af-4282-b567-66a4d880c3b4" containerName="istio-proxy"
Nov 02 01:56:40.790765 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.790658    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbdac3f5-20af-4282-b567-66a4d880c3b4" containerName="istio-proxy"
Nov 02 01:56:40.790765 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.790671    5639 cpu_manager.go:414] "containerMap: removing container" podUID="eba17c91-872f-468d-b14d-422f3fb73b4f" containerName="payload-pre-processing"
Nov 02 01:56:40.790765 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.790678    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eba17c91-872f-468d-b14d-422f3fb73b4f" containerName="payload-pre-processing"
Nov 02 01:56:40.790765 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.790724    5639 cpu_manager.go:414] "containerMap: removing container" podUID="c3b8159b-f16f-4a14-8077-255093cabef8" containerName="revoke-keys"
Nov 02 01:56:40.790765 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.790732    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3b8159b-f16f-4a14-8077-255093cabef8" containerName="revoke-keys"
Nov 02 01:56:40.790765 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.790742    5639 cpu_manager.go:414] "containerMap: removing container" podUID="b65549a1-6aa9-46a7-a2e7-1407ff736f44" containerName="payload-processing"
Nov 02 01:56:40.790765 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.790751    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b65549a1-6aa9-46a7-a2e7-1407ff736f44" containerName="payload-processing"
Nov 02 01:56:40.790765 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.790767    5639 cpu_manager.go:414] "containerMap: removing container" podUID="c4dec6db-d72f-45ce-802c-f54bb7f57e5c" containerName="istio-proxy"
Nov 02 01:56:40.791097 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.790774    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4dec6db-d72f-45ce-802c-f54bb7f57e5c" containerName="istio-proxy"
Nov 02 01:56:40.791097 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.790790    5639 cpu_manager.go:414] "containerMap: removing container" podUID="94b58fbf-2abd-4e6e-b40c-f7dcd8006713" containerName="istio-proxy"
Nov 02 01:56:40.791097 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.790797    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94b58fbf-2abd-4e6e-b40c-f7dcd8006713" containerName="istio-proxy"
Nov 02 01:56:40.791097 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.790959    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="94b58fbf-2abd-4e6e-b40c-f7dcd8006713" containerName="istio-proxy"
Nov 02 01:56:40.791097 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.790975    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4bfc70f-fa52-4876-9699-56040fd06e57" containerName="istio-proxy"
Nov 02 01:56:40.791097 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.790987    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4dec6db-d72f-45ce-802c-f54bb7f57e5c" containerName="istio-proxy"
Nov 02 01:56:40.791097 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.790999    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbdac3f5-20af-4282-b567-66a4d880c3b4" containerName="istio-proxy"
Nov 02 01:56:40.791097 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.791005    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="eba17c91-872f-468d-b14d-422f3fb73b4f" containerName="payload-pre-processing"
Nov 02 01:56:40.791097 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.791022    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3b8159b-f16f-4a14-8077-255093cabef8" containerName="revoke-keys"
Nov 02 01:56:40.791097 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.791030    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="b65549a1-6aa9-46a7-a2e7-1407ff736f44" containerName="payload-processing"
Nov 02 01:56:40.797536 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.797512    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs"
Nov 02 01:56:40.835972 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.835939    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs"]
Nov 02 01:56:40.899357 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.884121    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-218802-dockercfg-kmqrh\""
Nov 02 01:56:40.899357 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:40.884200    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-218802\""
Nov 02 01:56:41.018313 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:41.014988    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw"]
Nov 02 01:56:41.018313 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:41.016300    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4dec6db-d72f-45ce-802c-f54bb7f57e5c" containerName="istio-proxy"
Nov 02 01:56:41.077944 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:41.021071    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw"
Nov 02 01:56:41.077944 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:41.035678    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8815f4d6-021e-4c5f-9638-5c53d8470c5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs\" (UID: \"8815f4d6-021e-4c5f-9638-5c53d8470c5d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs"
Nov 02 01:56:41.077944 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:41.035769    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hcbz\" (UniqueName: \"kubernetes.io/projected/8815f4d6-021e-4c5f-9638-5c53d8470c5d-kube-api-access-9hcbz\") pod \"payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs\" (UID: \"8815f4d6-021e-4c5f-9638-5c53d8470c5d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs"
Nov 02 01:56:41.150673 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:41.142345    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67q72\" (UniqueName: \"kubernetes.io/projected/43fdb542-4aa2-47d6-9d83-f29f584cf771-kube-api-access-67q72\") pod \"payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw\" (UID: \"43fdb542-4aa2-47d6-9d83-f29f584cf771\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw"
Nov 02 01:56:41.150673 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:41.142401    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8815f4d6-021e-4c5f-9638-5c53d8470c5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs\" (UID: \"8815f4d6-021e-4c5f-9638-5c53d8470c5d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs"
Nov 02 01:56:41.150673 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:41.142471    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hcbz\" (UniqueName: \"kubernetes.io/projected/8815f4d6-021e-4c5f-9638-5c53d8470c5d-kube-api-access-9hcbz\") pod \"payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs\" (UID: \"8815f4d6-021e-4c5f-9638-5c53d8470c5d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs"
Nov 02 01:56:41.150673 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:41.142704    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43fdb542-4aa2-47d6-9d83-f29f584cf771-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw\" (UID: \"43fdb542-4aa2-47d6-9d83-f29f584cf771\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw"
Nov 02 01:56:41.150673 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:41.143724    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8815f4d6-021e-4c5f-9638-5c53d8470c5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs\" (UID: \"8815f4d6-021e-4c5f-9638-5c53d8470c5d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs"
Nov 02 01:56:41.155496 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:41.155468    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw"]
Nov 02 01:56:41.204880 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:41.204849    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hcbz\" (UniqueName: \"kubernetes.io/projected/8815f4d6-021e-4c5f-9638-5c53d8470c5d-kube-api-access-9hcbz\") pod \"payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs\" (UID: \"8815f4d6-021e-4c5f-9638-5c53d8470c5d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs"
Nov 02 01:56:41.244416 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:41.244381    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43fdb542-4aa2-47d6-9d83-f29f584cf771-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw\" (UID: \"43fdb542-4aa2-47d6-9d83-f29f584cf771\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw"
Nov 02 01:56:41.244564 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:41.244481    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67q72\" (UniqueName: \"kubernetes.io/projected/43fdb542-4aa2-47d6-9d83-f29f584cf771-kube-api-access-67q72\") pod \"payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw\" (UID: \"43fdb542-4aa2-47d6-9d83-f29f584cf771\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw"
Nov 02 01:56:41.245050 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:41.245032    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43fdb542-4aa2-47d6-9d83-f29f584cf771-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw\" (UID: \"43fdb542-4aa2-47d6-9d83-f29f584cf771\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw"
Nov 02 01:56:41.305125 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:41.305090    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67q72\" (UniqueName: \"kubernetes.io/projected/43fdb542-4aa2-47d6-9d83-f29f584cf771-kube-api-access-67q72\") pod \"payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw\" (UID: \"43fdb542-4aa2-47d6-9d83-f29f584cf771\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw"
Nov 02 01:56:41.414313 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:41.414197    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs"
Nov 02 01:56:41.462029 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:41.461994    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw"
Nov 02 01:56:41.794213 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:41.794180    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs"]
Nov 02 01:56:41.794962 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:56:41.794937    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8815f4d6_021e_4c5f_9638_5c53d8470c5d.slice/crio-67420e952da8e0df8f8abe84ac860a7646dab4b7e52fca870f66e8d55698fdf5 WatchSource:0}: Error finding container 67420e952da8e0df8f8abe84ac860a7646dab4b7e52fca870f66e8d55698fdf5: Status 404 returned error can't find the container with id 67420e952da8e0df8f8abe84ac860a7646dab4b7e52fca870f66e8d55698fdf5
Nov 02 01:56:41.894190 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:41.894165    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw"]
Nov 02 01:56:41.895117 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:56:41.895091    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43fdb542_4aa2_47d6_9d83_f29f584cf771.slice/crio-70f17dac9218c7e1fef6f832764d4c19a820edcc3a20c1f1299fd2e5f99cd3bb WatchSource:0}: Error finding container 70f17dac9218c7e1fef6f832764d4c19a820edcc3a20c1f1299fd2e5f99cd3bb: Status 404 returned error can't find the container with id 70f17dac9218c7e1fef6f832764d4c19a820edcc3a20c1f1299fd2e5f99cd3bb
Nov 02 01:56:42.556901 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:42.556861    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw" event={"ID":"43fdb542-4aa2-47d6-9d83-f29f584cf771","Type":"ContainerStarted","Data":"68ed8a26ab82b43d6a6ab2a3ee684befae8c8a35deb25c862a084f135ab968af"}
Nov 02 01:56:42.557105 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:42.556923    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw" event={"ID":"43fdb542-4aa2-47d6-9d83-f29f584cf771","Type":"ContainerStarted","Data":"70f17dac9218c7e1fef6f832764d4c19a820edcc3a20c1f1299fd2e5f99cd3bb"}
Nov 02 01:56:42.557105 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:42.556952    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw"
Nov 02 01:56:42.558801 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:42.558772    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs" event={"ID":"8815f4d6-021e-4c5f-9638-5c53d8470c5d","Type":"ContainerStarted","Data":"3049b3a58f82c4549afdc6cc77d5b89cebd558dedb9f5d9f34dd6c19116ded74"}
Nov 02 01:56:42.558932 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:42.558809    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs" event={"ID":"8815f4d6-021e-4c5f-9638-5c53d8470c5d","Type":"ContainerStarted","Data":"67420e952da8e0df8f8abe84ac860a7646dab4b7e52fca870f66e8d55698fdf5"}
Nov 02 01:56:42.558932 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:42.558882    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs"
Nov 02 01:56:42.675150 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:42.675095    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw" podStartSLOduration=2.6750833590000003 podStartE2EDuration="2.675083359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:56:42.672145283 +0000 UTC m=+1867.575827399" watchObservedRunningTime="2026-11-02 01:56:42.675083359 +0000 UTC m=+1867.578765492"
Nov 02 01:56:50.020719 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.020669    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs" podStartSLOduration=10.020654294 podStartE2EDuration="10.020654294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:56:42.805645655 +0000 UTC m=+1867.709327770" watchObservedRunningTime="2026-11-02 01:56:50.020654294 +0000 UTC m=+1874.924336408"
Nov 02 01:56:50.021314 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.021272    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"]
Nov 02 01:56:50.025906 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.025890    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.143959 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.119168    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-19436b-openshift-default-dockercfg-mxlf6\""
Nov 02 01:56:50.335409 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.156315    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"]
Nov 02 01:56:50.335827 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.316028    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-credential-socket\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.336139 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.336106    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8da016-d523-4efd-91de-aae563f7f1a6-istiod-ca-cert\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.336497 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.336452    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-workload-socket\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.336713 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.336694    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-podinfo\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.336870 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.336848    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-token\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.337071 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.337051    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-workload-certs\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.337219 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.337201    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x8p8\" (UniqueName: \"kubernetes.io/projected/5d8da016-d523-4efd-91de-aae563f7f1a6-kube-api-access-9x8p8\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.337503 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.337476    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-envoy\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.337692 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.337665    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-data\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.461869 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.461816    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-workload-socket\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.462183 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.462158    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-podinfo\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.462356 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.462338    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-token\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.462627 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.462609    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-workload-certs\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.462823 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.462759    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x8p8\" (UniqueName: \"kubernetes.io/projected/5d8da016-d523-4efd-91de-aae563f7f1a6-kube-api-access-9x8p8\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.463028 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.463010    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-envoy\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.463195 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.463174    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-data\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.463543 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.463491    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-credential-socket\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.463772 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.463743    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8da016-d523-4efd-91de-aae563f7f1a6-istiod-ca-cert\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.464921 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.464888    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8da016-d523-4efd-91de-aae563f7f1a6-istiod-ca-cert\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.465362 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.465342    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-workload-socket\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.485515 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.470660    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-workload-certs\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.485515 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.471261    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-credential-socket\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.504488 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.498502    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-data\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.504488 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.498612    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-envoy\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.522071 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.509987    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-podinfo\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.575470 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.573365    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x8p8\" (UniqueName: \"kubernetes.io/projected/5d8da016-d523-4efd-91de-aae563f7f1a6-kube-api-access-9x8p8\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.606889 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.605910    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-token\") pod \"e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") " pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.641591 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.641555    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:50.982322 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.982254    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4"]
Nov 02 01:56:50.994544 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:50.994511    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4"
Nov 02 01:56:51.029970 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.029942    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ad3b96d-b251-48d6-8b20-5e6f73f48448-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4\" (UID: \"8ad3b96d-b251-48d6-8b20-5e6f73f48448\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4"
Nov 02 01:56:51.030454 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.030071    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97rh8\" (UniqueName: \"kubernetes.io/projected/8ad3b96d-b251-48d6-8b20-5e6f73f48448-kube-api-access-97rh8\") pod \"payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4\" (UID: \"8ad3b96d-b251-48d6-8b20-5e6f73f48448\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4"
Nov 02 01:56:51.043135 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.043106    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4"]
Nov 02 01:56:51.060304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.055539    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-bb0464\""
Nov 02 01:56:51.060304 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.055713    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-bb0464-dockercfg-9zh8n\""
Nov 02 01:56:51.234875 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.234792    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ad3b96d-b251-48d6-8b20-5e6f73f48448-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4\" (UID: \"8ad3b96d-b251-48d6-8b20-5e6f73f48448\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4"
Nov 02 01:56:51.235043 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.234893    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97rh8\" (UniqueName: \"kubernetes.io/projected/8ad3b96d-b251-48d6-8b20-5e6f73f48448-kube-api-access-97rh8\") pod \"payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4\" (UID: \"8ad3b96d-b251-48d6-8b20-5e6f73f48448\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4"
Nov 02 01:56:51.319300 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.302663    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ad3b96d-b251-48d6-8b20-5e6f73f48448-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4\" (UID: \"8ad3b96d-b251-48d6-8b20-5e6f73f48448\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4"
Nov 02 01:56:51.324157 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.323399    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97rh8\" (UniqueName: \"kubernetes.io/projected/8ad3b96d-b251-48d6-8b20-5e6f73f48448-kube-api-access-97rh8\") pod \"payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4\" (UID: \"8ad3b96d-b251-48d6-8b20-5e6f73f48448\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4"
Nov 02 01:56:51.336108 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.336069    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4"
Nov 02 01:56:51.364434 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.364379    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:56:51.364545 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.364476    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:56:51.364584 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.364548    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:56:51.366516 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.366487    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92"]
Nov 02 01:56:51.373209 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.373185    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92"]
Nov 02 01:56:51.373327 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.373268    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92"
Nov 02 01:56:51.374023 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.373995    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"]
Nov 02 01:56:51.459262 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.459229    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8spjh\" (UniqueName: \"kubernetes.io/projected/94012b0a-9356-41bc-8235-9a64f11e8e39-kube-api-access-8spjh\") pod \"payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92\" (UID: \"94012b0a-9356-41bc-8235-9a64f11e8e39\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92"
Nov 02 01:56:51.459738 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.459707    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94012b0a-9356-41bc-8235-9a64f11e8e39-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92\" (UID: \"94012b0a-9356-41bc-8235-9a64f11e8e39\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92"
Nov 02 01:56:51.561719 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.561623    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94012b0a-9356-41bc-8235-9a64f11e8e39-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92\" (UID: \"94012b0a-9356-41bc-8235-9a64f11e8e39\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92"
Nov 02 01:56:51.561899 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.561874    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8spjh\" (UniqueName: \"kubernetes.io/projected/94012b0a-9356-41bc-8235-9a64f11e8e39-kube-api-access-8spjh\") pod \"payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92\" (UID: \"94012b0a-9356-41bc-8235-9a64f11e8e39\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92"
Nov 02 01:56:51.562887 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.562848    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94012b0a-9356-41bc-8235-9a64f11e8e39-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92\" (UID: \"94012b0a-9356-41bc-8235-9a64f11e8e39\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92"
Nov 02 01:56:51.623214 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.623183    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8spjh\" (UniqueName: \"kubernetes.io/projected/94012b0a-9356-41bc-8235-9a64f11e8e39-kube-api-access-8spjh\") pod \"payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92\" (UID: \"94012b0a-9356-41bc-8235-9a64f11e8e39\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92"
Nov 02 01:56:51.633607 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.633572    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx" event={"ID":"5d8da016-d523-4efd-91de-aae563f7f1a6","Type":"ContainerStarted","Data":"325a54703dad3b6d5437e231b5ef8bcdc8902b51ca8435ee5ee39c9f432b6603"}
Nov 02 01:56:51.633769 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.633633    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx" event={"ID":"5d8da016-d523-4efd-91de-aae563f7f1a6","Type":"ContainerStarted","Data":"71677e740232da8a6b130ca366fee1c45352d8954873124c6f43745a4deaf066"}
Nov 02 01:56:51.642043 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.642018    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:51.687265 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.687234    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv"]
Nov 02 01:56:51.688994 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.688975    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92"
Nov 02 01:56:51.692025 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.692000    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-59842dd7-5rkxv"]
Nov 02 01:56:51.760516 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.759950    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4"]
Nov 02 01:56:51.775541 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:51.773524    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx" podStartSLOduration=2.773504561 podStartE2EDuration="2.773504561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:56:51.772855718 +0000 UTC m=+1876.676537845" watchObservedRunningTime="2026-11-02 01:56:51.773504561 +0000 UTC m=+1876.677186677"
Nov 02 01:56:52.143220 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:52.143188    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92"]
Nov 02 01:56:52.144643 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:56:52.144616    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94012b0a_9356_41bc_8235_9a64f11e8e39.slice/crio-ffe13f28eb6d481a53fd6bf3903313a004bf4dcc539c322a57bfd8482b814dfb WatchSource:0}: Error finding container ffe13f28eb6d481a53fd6bf3903313a004bf4dcc539c322a57bfd8482b814dfb: Status 404 returned error can't find the container with id ffe13f28eb6d481a53fd6bf3903313a004bf4dcc539c322a57bfd8482b814dfb
Nov 02 01:56:52.642482 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:52.642451    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4" event={"ID":"8ad3b96d-b251-48d6-8b20-5e6f73f48448","Type":"ContainerStarted","Data":"62707b9e9f128e0021f2e2002904b0eea907694eb405e51f3850c383eb8159fb"}
Nov 02 01:56:52.642695 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:52.642516    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4" event={"ID":"8ad3b96d-b251-48d6-8b20-5e6f73f48448","Type":"ContainerStarted","Data":"1a2015c57b80263860016a025b1e7d26627a82db6def9448ac805bc506d1bec8"}
Nov 02 01:56:52.642695 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:52.642546    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4"
Nov 02 01:56:52.644571 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:52.644535    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92" event={"ID":"94012b0a-9356-41bc-8235-9a64f11e8e39","Type":"ContainerStarted","Data":"d01fd69e4214f80db61711a21b8d36e9a60858d27e2840eb777215641a6960d9"}
Nov 02 01:56:52.644696 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:52.644596    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92" event={"ID":"94012b0a-9356-41bc-8235-9a64f11e8e39","Type":"ContainerStarted","Data":"ffe13f28eb6d481a53fd6bf3903313a004bf4dcc539c322a57bfd8482b814dfb"}
Nov 02 01:56:52.648244 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:52.648222    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:52.760340 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:52.760251    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4" podStartSLOduration=2.760236188 podStartE2EDuration="2.760236188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:56:52.75932039 +0000 UTC m=+1877.663002526" watchObservedRunningTime="2026-11-02 01:56:52.760236188 +0000 UTC m=+1877.663918301"
Nov 02 01:56:53.038347 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:53.038213    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92" podStartSLOduration=2.038197876 podStartE2EDuration="2.038197876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:56:53.035100139 +0000 UTC m=+1877.938782252" watchObservedRunningTime="2026-11-02 01:56:53.038197876 +0000 UTC m=+1877.941880062"
Nov 02 01:56:53.244655 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:53.244613    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3b8159b-f16f-4a14-8077-255093cabef8" path="/var/lib/kubelet/pods/c3b8159b-f16f-4a14-8077-255093cabef8/volumes"
Nov 02 01:56:53.570003 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:53.569974    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw"
Nov 02 01:56:53.570223 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:53.570120    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs"
Nov 02 01:56:53.655398 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:53.655366    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:53.655398 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:53.655447    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92"
Nov 02 01:56:53.657065 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:53.657044    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:56:57.423235 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:56:57.423197    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:56:58.530557 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:58.530522    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:56:59.362618 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:59.362580    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"]
Nov 02 01:56:59.362855 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:56:59.362829    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d" podUID="86b56f9b-4c16-412c-b170-76133ab7abb2" containerName="istio-proxy" containerID="cri-o://561ab8df5652445fbd5eb4a2abb2df89dfcba47c475976070e4a24ce46ecd4cd" gracePeriod=30
Nov 02 01:57:03.658934 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:03.658901    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4"
Nov 02 01:57:04.732640 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:04.732549    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92"
Nov 02 01:57:04.743246 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:04.743218    5639 generic.go:356] "Generic (PLEG): container finished" podID="86b56f9b-4c16-412c-b170-76133ab7abb2" containerID="561ab8df5652445fbd5eb4a2abb2df89dfcba47c475976070e4a24ce46ecd4cd" exitCode=0
Nov 02 01:57:04.743414 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:04.743259    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d" event={"ID":"86b56f9b-4c16-412c-b170-76133ab7abb2","Type":"ContainerDied","Data":"561ab8df5652445fbd5eb4a2abb2df89dfcba47c475976070e4a24ce46ecd4cd"}
Nov 02 01:57:04.743414 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:04.743325    5639 scope.go:122] "RemoveContainer" containerID="fbeb990c66f31886a87770784922ccdee559e57d9929090671b9a8fb659e3375"
Nov 02 01:57:04.801603 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:04.801495    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:57:04.998378 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:04.944722    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"]
Nov 02 01:57:05.039306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:04.962235    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-envoy\") pod \"86b56f9b-4c16-412c-b170-76133ab7abb2\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") "
Nov 02 01:57:05.049338 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.039672    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-workload-socket\") pod \"86b56f9b-4c16-412c-b170-76133ab7abb2\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") "
Nov 02 01:57:05.089303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.049689    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-workload-certs\") pod \"86b56f9b-4c16-412c-b170-76133ab7abb2\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") "
Nov 02 01:57:05.089921 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.089887    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86b56f9b-4c16-412c-b170-76133ab7abb2-kube-api-access-pbk9p\" (UniqueName: \"kubernetes.io/projected/86b56f9b-4c16-412c-b170-76133ab7abb2-kube-api-access-pbk9p\") pod \"86b56f9b-4c16-412c-b170-76133ab7abb2\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") "
Nov 02 01:57:05.113313 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.090194    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-credential-socket\") pod \"86b56f9b-4c16-412c-b170-76133ab7abb2\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") "
Nov 02 01:57:05.128300 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.060394    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-workload-socket" pod "86b56f9b-4c16-412c-b170-76133ab7abb2" (UID: "86b56f9b-4c16-412c-b170-76133ab7abb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:05.128300 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.060815    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-workload-certs" pod "86b56f9b-4c16-412c-b170-76133ab7abb2" (UID: "86b56f9b-4c16-412c-b170-76133ab7abb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:05.128300 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.091336    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-credential-socket" pod "86b56f9b-4c16-412c-b170-76133ab7abb2" (UID: "86b56f9b-4c16-412c-b170-76133ab7abb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:05.128300 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.105145    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b56f9b-4c16-412c-b170-76133ab7abb2-kube-api-access-pbk9p" pod "86b56f9b-4c16-412c-b170-76133ab7abb2" (UID: "86b56f9b-4c16-412c-b170-76133ab7abb2"). InnerVolumeSpecName "kube-api-access-pbk9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:05.128300 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.108206    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-envoy" pod "86b56f9b-4c16-412c-b170-76133ab7abb2" (UID: "86b56f9b-4c16-412c-b170-76133ab7abb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:05.128300 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.042422    5639 cpu_manager.go:414] "containerMap: removing container" podUID="86b56f9b-4c16-412c-b170-76133ab7abb2" containerName="istio-proxy"
Nov 02 01:57:05.128300 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.115597    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86b56f9b-4c16-412c-b170-76133ab7abb2" containerName="istio-proxy"
Nov 02 01:57:05.128300 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.115668    5639 cpu_manager.go:414] "containerMap: removing container" podUID="86b56f9b-4c16-412c-b170-76133ab7abb2" containerName="istio-proxy"
Nov 02 01:57:05.128300 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.115678    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86b56f9b-4c16-412c-b170-76133ab7abb2" containerName="istio-proxy"
Nov 02 01:57:05.128300 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.115951    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="86b56f9b-4c16-412c-b170-76133ab7abb2" containerName="istio-proxy"
Nov 02 01:57:05.218814 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.218786    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86b56f9b-4c16-412c-b170-76133ab7abb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b56f9b-4c16-412c-b170-76133ab7abb2-istiod-ca-cert\") pod \"86b56f9b-4c16-412c-b170-76133ab7abb2\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") "
Nov 02 01:57:05.219101 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.219084    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-data\") pod \"86b56f9b-4c16-412c-b170-76133ab7abb2\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") "
Nov 02 01:57:05.219310 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.219295    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-podinfo\") pod \"86b56f9b-4c16-412c-b170-76133ab7abb2\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") "
Nov 02 01:57:05.219438 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.219409    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-token\" (UniqueName: \"kubernetes.io/projected/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-token\") pod \"86b56f9b-4c16-412c-b170-76133ab7abb2\" (UID: \"86b56f9b-4c16-412c-b170-76133ab7abb2\") "
Nov 02 01:57:05.220585 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.220525    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:05.237837 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.237809    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:05.256581 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.256435    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:05.256942 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.256924    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:05.257048 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.257036    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbk9p\" (UniqueName: \"kubernetes.io/projected/86b56f9b-4c16-412c-b170-76133ab7abb2-kube-api-access-pbk9p\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:05.257133 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.234069    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-data" pod "86b56f9b-4c16-412c-b170-76133ab7abb2" (UID: "86b56f9b-4c16-412c-b170-76133ab7abb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:05.257213 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.234475    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86b56f9b-4c16-412c-b170-76133ab7abb2-istiod-ca-cert" pod "86b56f9b-4c16-412c-b170-76133ab7abb2" (UID: "86b56f9b-4c16-412c-b170-76133ab7abb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:57:05.283306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.274516    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-token" pod "86b56f9b-4c16-412c-b170-76133ab7abb2" (UID: "86b56f9b-4c16-412c-b170-76133ab7abb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:05.283306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.274614    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-podinfo" pod "86b56f9b-4c16-412c-b170-76133ab7abb2" (UID: "86b56f9b-4c16-412c-b170-76133ab7abb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:57:05.309046 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.306834    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.322066 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.322038    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"]
Nov 02 01:57:05.358983 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.358935    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-data\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.373352 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.365353    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-podinfo\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.373723 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.373679    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-credential-socket\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.373954 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.373936    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-token\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.385630 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.376024    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2mcw\" (UniqueName: \"kubernetes.io/projected/b2e9988f-011a-4a80-ad65-d54e5266e8dd-kube-api-access-k2mcw\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.412332 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.386021    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-workload-socket\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.414387 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.414354    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istiod-ca-cert\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.414642 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.414623    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-workload-certs\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.414864 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.414845    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-envoy\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.415671 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.367856    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-109b07-openshift-default-dockercfg-s6pbb\""
Nov 02 01:57:05.416050 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.416017    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b56f9b-4c16-412c-b170-76133ab7abb2-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:05.416216 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.416179    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:05.416361 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.416348    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:05.416512 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.416500    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b56f9b-4c16-412c-b170-76133ab7abb2-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:05.517939 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.517845    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-workload-socket\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.517939 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.517908    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istiod-ca-cert\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.518442 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.517955    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-workload-certs\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.518442 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.517996    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-envoy\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.518442 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.518115    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-data\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.518442 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.518231    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-podinfo\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.518442 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.518323    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-credential-socket\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.518442 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.518370    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-workload-socket\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.518442 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.518407    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-token\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.518867 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.518462    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2mcw\" (UniqueName: \"kubernetes.io/projected/b2e9988f-011a-4a80-ad65-d54e5266e8dd-kube-api-access-k2mcw\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.519518 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.519133    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istiod-ca-cert\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.519518 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.519202    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-credential-socket\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.519518 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.519318    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-workload-certs\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.519518 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.519482    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-data\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.521597 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.521576    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-podinfo\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.522511 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.522491    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-envoy\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.579747 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.579715    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2mcw\" (UniqueName: \"kubernetes.io/projected/b2e9988f-011a-4a80-ad65-d54e5266e8dd-kube-api-access-k2mcw\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.580494 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.580468    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-token\") pod \"e2e-api-a-109b07-openshift-default-54786bf495-brglf\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") " pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.629586 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.629552    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:05.760852 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.760824    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"
Nov 02 01:57:05.761325 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.760823    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d" event={"ID":"86b56f9b-4c16-412c-b170-76133ab7abb2","Type":"ContainerDied","Data":"5ceb8f7ba6d1ad9a9e3f4e14a824060d79a352c6f62f78b25951b48cb8173484"}
Nov 02 01:57:05.761325 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:05.760968    5639 scope.go:122] "RemoveContainer" containerID="561ab8df5652445fbd5eb4a2abb2df89dfcba47c475976070e4a24ce46ecd4cd"
Nov 02 01:57:06.001967 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:06.001907    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"]
Nov 02 01:57:06.006825 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:06.006797    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-59842dd7-openshift-default-84668d95bf-zmc7d"]
Nov 02 01:57:06.067703 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:06.067674    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"]
Nov 02 01:57:06.068179 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:57:06.068153    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e9988f_011a_4a80_ad65_d54e5266e8dd.slice/crio-adb52972c3ed260414b20cb8b5f7df8e04e023b2d1cc9ec96dac2d2346bcfc43 WatchSource:0}: Error finding container adb52972c3ed260414b20cb8b5f7df8e04e023b2d1cc9ec96dac2d2346bcfc43: Status 404 returned error can't find the container with id adb52972c3ed260414b20cb8b5f7df8e04e023b2d1cc9ec96dac2d2346bcfc43
Nov 02 01:57:06.071097 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:06.071064    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:57:06.071184 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:06.071131    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:57:06.071184 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:06.071161    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:57:06.773153 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:06.773114    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf" event={"ID":"b2e9988f-011a-4a80-ad65-d54e5266e8dd","Type":"ContainerStarted","Data":"ed62dd24cfa8cc54208107256e01590ea5f2a6782a2a17b58ee356598ee85378"}
Nov 02 01:57:06.773646 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:06.773188    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf" event={"ID":"b2e9988f-011a-4a80-ad65-d54e5266e8dd","Type":"ContainerStarted","Data":"adb52972c3ed260414b20cb8b5f7df8e04e023b2d1cc9ec96dac2d2346bcfc43"}
Nov 02 01:57:06.930151 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:06.930098    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf" podStartSLOduration=2.930083654 podStartE2EDuration="2.930083654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:57:06.892944767 +0000 UTC m=+1891.796626892" watchObservedRunningTime="2026-11-02 01:57:06.930083654 +0000 UTC m=+1891.833765767"
Nov 02 01:57:07.222056 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:07.222018    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86b56f9b-4c16-412c-b170-76133ab7abb2" path="/var/lib/kubelet/pods/86b56f9b-4c16-412c-b170-76133ab7abb2/volumes"
Nov 02 01:57:07.629800 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:07.629765    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:07.635914 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:07.635888    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:07.783722 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:07.783691    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:07.785010 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:07.784992    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:57:09.453777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:09.450753    5639 patch_prober.go:29] interesting pod/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": read tcp 10.134.0.2:45818->10.134.0.171:15021: read: connection reset by peer" start-of-body=
Nov 02 01:57:09.453777 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:09.450832    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs" podUID="7542d266-7f8c-474a-a8dc-9848376e3993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": read tcp 10.134.0.2:45818->10.134.0.171:15021: read: connection reset by peer"
Nov 02 01:57:09.800402 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:09.800264    5639 generic.go:356] "Generic (PLEG): container finished" podID="7542d266-7f8c-474a-a8dc-9848376e3993" containerID="e662c189e08dcec93fdbe7cc8b5a01617477d671284cbf448c38a4df9f102569" exitCode=0
Nov 02 01:57:09.800402 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:09.800339    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs" event={"ID":"7542d266-7f8c-474a-a8dc-9848376e3993","Type":"ContainerDied","Data":"e662c189e08dcec93fdbe7cc8b5a01617477d671284cbf448c38a4df9f102569"}
Nov 02 01:57:09.801068 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:09.801046    5639 scope.go:122] "RemoveContainer" containerID="e662c189e08dcec93fdbe7cc8b5a01617477d671284cbf448c38a4df9f102569"
Nov 02 01:57:09.802593 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:09.802564    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:57:09.802705 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:09.802631    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:57:09.802705 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:09.802661    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:57:10.813186 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:10.813150    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs" event={"ID":"7542d266-7f8c-474a-a8dc-9848376e3993","Type":"ContainerStarted","Data":"0601eb1677436e92301b075d8cebd6c527170d03ffaa0895b231b2962dac6305"}
Nov 02 01:57:11.605729 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:11.605698    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:57:11.605926 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:11.605867    5639 patch_prober.go:29] interesting pod/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Nov 02 01:57:11.605985 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:11.605930    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs" podUID="7542d266-7f8c-474a-a8dc-9848376e3993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Nov 02 01:57:12.607968 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:12.605349    5639 patch_prober.go:29] interesting pod/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Nov 02 01:57:12.607968 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:12.605475    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs" podUID="7542d266-7f8c-474a-a8dc-9848376e3993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Nov 02 01:57:13.605219 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:13.605184    5639 patch_prober.go:29] interesting pod/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Nov 02 01:57:13.605436 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:13.605242    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs" podUID="7542d266-7f8c-474a-a8dc-9848376e3993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Nov 02 01:57:14.604845 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:14.604814    5639 patch_prober.go:29] interesting pod/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Nov 02 01:57:14.605242 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:14.604860    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs" podUID="7542d266-7f8c-474a-a8dc-9848376e3993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Nov 02 01:57:15.292949 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:15.292915    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"]
Nov 02 01:57:15.293182 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:15.293113    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx" podUID="cd05aad1-3b5f-43bd-988e-fce23728a36c" containerName="istio-proxy" containerID="cri-o://933c61e1d77c5c3d7b112435f828d16379c5e7e219797a482cc5c57783722be3" gracePeriod=30
Nov 02 01:57:15.341566 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:15.341489    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"]
Nov 02 01:57:15.341733 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:15.341694    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r" podUID="4e35e13f-b6eb-4a4c-8142-f54a5c621a97" containerName="istio-proxy" containerID="cri-o://9dc87c280ab43af3fd72d7b2532fcbc0505d9fc0addfd1a3bab0e42853db8e3a" gracePeriod=30
Nov 02 01:57:15.604903 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:15.604874    5639 patch_prober.go:29] interesting pod/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Nov 02 01:57:15.605345 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:15.604926    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs" podUID="7542d266-7f8c-474a-a8dc-9848376e3993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Nov 02 01:57:16.609347 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:16.609317    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:57:16.609789 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:16.609734    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:57:16.610233 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:16.610216    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 01:57:18.700362 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:18.700319    5639 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:57:18.700778 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:18.700397    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx" podUID="cd05aad1-3b5f-43bd-988e-fce23728a36c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:57:20.646530 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.646504    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:57:20.691297 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.691259    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:57:20.743893 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.743857    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-workload-certs\") pod \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") "
Nov 02 01:57:20.744083 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.743902    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-data\") pod \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") "
Nov 02 01:57:20.744083 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.743963    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-credential-socket\") pod \"cd05aad1-3b5f-43bd-988e-fce23728a36c\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") "
Nov 02 01:57:20.744083 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.743984    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-token\" (UniqueName: \"kubernetes.io/projected/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-token\") pod \"cd05aad1-3b5f-43bd-988e-fce23728a36c\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") "
Nov 02 01:57:20.744083 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.744003    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd05aad1-3b5f-43bd-988e-fce23728a36c-kube-api-access-gz5mg\" (UniqueName: \"kubernetes.io/projected/cd05aad1-3b5f-43bd-988e-fce23728a36c-kube-api-access-gz5mg\") pod \"cd05aad1-3b5f-43bd-988e-fce23728a36c\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") "
Nov 02 01:57:20.744363 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.744157    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-token\" (UniqueName: \"kubernetes.io/projected/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-token\") pod \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") "
Nov 02 01:57:20.744363 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.744197    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-workload-certs" pod "4e35e13f-b6eb-4a4c-8142-f54a5c621a97" (UID: "4e35e13f-b6eb-4a4c-8142-f54a5c621a97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:20.744363 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.744217    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd05aad1-3b5f-43bd-988e-fce23728a36c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd05aad1-3b5f-43bd-988e-fce23728a36c-istiod-ca-cert\") pod \"cd05aad1-3b5f-43bd-988e-fce23728a36c\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") "
Nov 02 01:57:20.744363 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.744266    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-envoy\") pod \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") "
Nov 02 01:57:20.744363 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.744272    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-data" pod "4e35e13f-b6eb-4a4c-8142-f54a5c621a97" (UID: "4e35e13f-b6eb-4a4c-8142-f54a5c621a97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:20.744363 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.744312    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-credential-socket" pod "cd05aad1-3b5f-43bd-988e-fce23728a36c" (UID: "cd05aad1-3b5f-43bd-988e-fce23728a36c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:20.744363 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.744354    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-workload-certs\") pod \"cd05aad1-3b5f-43bd-988e-fce23728a36c\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") "
Nov 02 01:57:20.744781 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.744380    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-data\") pod \"cd05aad1-3b5f-43bd-988e-fce23728a36c\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") "
Nov 02 01:57:20.744781 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.744408    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-podinfo\") pod \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") "
Nov 02 01:57:20.744781 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.744454    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-credential-socket\") pod \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") "
Nov 02 01:57:20.744781 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.744536    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-envoy\") pod \"cd05aad1-3b5f-43bd-988e-fce23728a36c\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") "
Nov 02 01:57:20.744781 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.744587    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istiod-ca-cert\") pod \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") "
Nov 02 01:57:20.744781 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.744620    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-kube-api-access-zvqkf\" (UniqueName: \"kubernetes.io/projected/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-kube-api-access-zvqkf\") pod \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") "
Nov 02 01:57:20.744781 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.744653    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-workload-certs" pod "cd05aad1-3b5f-43bd-988e-fce23728a36c" (UID: "cd05aad1-3b5f-43bd-988e-fce23728a36c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:20.744781 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.744664    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-workload-socket\") pod \"cd05aad1-3b5f-43bd-988e-fce23728a36c\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") "
Nov 02 01:57:20.744781 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.744667    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd05aad1-3b5f-43bd-988e-fce23728a36c-istiod-ca-cert" pod "cd05aad1-3b5f-43bd-988e-fce23728a36c" (UID: "cd05aad1-3b5f-43bd-988e-fce23728a36c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:57:20.744781 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.744737    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-podinfo\") pod \"cd05aad1-3b5f-43bd-988e-fce23728a36c\" (UID: \"cd05aad1-3b5f-43bd-988e-fce23728a36c\") "
Nov 02 01:57:20.745487 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.744813    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-workload-socket\") pod \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\" (UID: \"4e35e13f-b6eb-4a4c-8142-f54a5c621a97\") "
Nov 02 01:57:20.745487 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.744892    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-credential-socket" pod "4e35e13f-b6eb-4a4c-8142-f54a5c621a97" (UID: "4e35e13f-b6eb-4a4c-8142-f54a5c621a97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:20.745487 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.745243    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-data" pod "cd05aad1-3b5f-43bd-988e-fce23728a36c" (UID: "cd05aad1-3b5f-43bd-988e-fce23728a36c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:20.745487 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.745356    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-workload-socket" pod "cd05aad1-3b5f-43bd-988e-fce23728a36c" (UID: "cd05aad1-3b5f-43bd-988e-fce23728a36c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:20.745851 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.745716    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.745851 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.745741    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.745851 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.745755    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd05aad1-3b5f-43bd-988e-fce23728a36c-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.745851 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.745768    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.745851 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.745781    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.745851 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.745794    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.745851 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.745809    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.745851 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.745813    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-workload-socket" pod "4e35e13f-b6eb-4a4c-8142-f54a5c621a97" (UID: "4e35e13f-b6eb-4a4c-8142-f54a5c621a97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:20.745851 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.745824    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.746632 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.746040    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istiod-ca-cert" pod "4e35e13f-b6eb-4a4c-8142-f54a5c621a97" (UID: "4e35e13f-b6eb-4a4c-8142-f54a5c621a97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:57:20.747739 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.747641    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-kube-api-access-zvqkf" pod "4e35e13f-b6eb-4a4c-8142-f54a5c621a97" (UID: "4e35e13f-b6eb-4a4c-8142-f54a5c621a97"). InnerVolumeSpecName "kube-api-access-zvqkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:20.747739 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.747690    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-envoy" pod "4e35e13f-b6eb-4a4c-8142-f54a5c621a97" (UID: "4e35e13f-b6eb-4a4c-8142-f54a5c621a97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:20.747739 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.747706    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-envoy" pod "cd05aad1-3b5f-43bd-988e-fce23728a36c" (UID: "cd05aad1-3b5f-43bd-988e-fce23728a36c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:20.747739 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.747711    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-token" pod "cd05aad1-3b5f-43bd-988e-fce23728a36c" (UID: "cd05aad1-3b5f-43bd-988e-fce23728a36c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:20.747739 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.747722    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd05aad1-3b5f-43bd-988e-fce23728a36c-kube-api-access-gz5mg" pod "cd05aad1-3b5f-43bd-988e-fce23728a36c" (UID: "cd05aad1-3b5f-43bd-988e-fce23728a36c"). InnerVolumeSpecName "kube-api-access-gz5mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:20.748046 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.747883    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-token" pod "4e35e13f-b6eb-4a4c-8142-f54a5c621a97" (UID: "4e35e13f-b6eb-4a4c-8142-f54a5c621a97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:20.748176 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.748158    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-podinfo" pod "4e35e13f-b6eb-4a4c-8142-f54a5c621a97" (UID: "4e35e13f-b6eb-4a4c-8142-f54a5c621a97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:57:20.748496 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.748473    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-podinfo" pod "cd05aad1-3b5f-43bd-988e-fce23728a36c" (UID: "cd05aad1-3b5f-43bd-988e-fce23728a36c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:57:20.848042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.847964    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.848042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.848000    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.848042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.848013    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.848042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.848023    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.848042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.848032    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvqkf\" (UniqueName: \"kubernetes.io/projected/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-kube-api-access-zvqkf\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.848042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.848043    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.848397 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.848052    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.848397 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.848060    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd05aad1-3b5f-43bd-988e-fce23728a36c-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.848397 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.848068    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz5mg\" (UniqueName: \"kubernetes.io/projected/cd05aad1-3b5f-43bd-988e-fce23728a36c-kube-api-access-gz5mg\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.848397 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.848078    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e35e13f-b6eb-4a4c-8142-f54a5c621a97-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.893089 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.893056    5639 generic.go:356] "Generic (PLEG): container finished" podID="cd05aad1-3b5f-43bd-988e-fce23728a36c" containerID="933c61e1d77c5c3d7b112435f828d16379c5e7e219797a482cc5c57783722be3" exitCode=0
Nov 02 01:57:20.893309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.893101    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"
Nov 02 01:57:20.893309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.893130    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx" event={"ID":"cd05aad1-3b5f-43bd-988e-fce23728a36c","Type":"ContainerDied","Data":"933c61e1d77c5c3d7b112435f828d16379c5e7e219797a482cc5c57783722be3"}
Nov 02 01:57:20.893309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.893177    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx" event={"ID":"cd05aad1-3b5f-43bd-988e-fce23728a36c","Type":"ContainerDied","Data":"becba962892b10a8df494b74fd5d3e93de9bdcc73d3fb9fa8e68fef36af9077c"}
Nov 02 01:57:20.893309 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.893206    5639 scope.go:122] "RemoveContainer" containerID="933c61e1d77c5c3d7b112435f828d16379c5e7e219797a482cc5c57783722be3"
Nov 02 01:57:20.894724 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.894705    5639 generic.go:356] "Generic (PLEG): container finished" podID="4e35e13f-b6eb-4a4c-8142-f54a5c621a97" containerID="9dc87c280ab43af3fd72d7b2532fcbc0505d9fc0addfd1a3bab0e42853db8e3a" exitCode=0
Nov 02 01:57:20.894836 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.894758    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"
Nov 02 01:57:20.894836 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.894779    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r" event={"ID":"4e35e13f-b6eb-4a4c-8142-f54a5c621a97","Type":"ContainerDied","Data":"9dc87c280ab43af3fd72d7b2532fcbc0505d9fc0addfd1a3bab0e42853db8e3a"}
Nov 02 01:57:20.894933 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.894833    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r" event={"ID":"4e35e13f-b6eb-4a4c-8142-f54a5c621a97","Type":"ContainerDied","Data":"26414b2a462bd223de39f38b5ae0b0cedd89a3d4d55df044bfe1c9fc5ecbb194"}
Nov 02 01:57:20.910645 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.910623    5639 scope.go:122] "RemoveContainer" containerID="933c61e1d77c5c3d7b112435f828d16379c5e7e219797a482cc5c57783722be3"
Nov 02 01:57:20.910980 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:57:20.910958    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933c61e1d77c5c3d7b112435f828d16379c5e7e219797a482cc5c57783722be3\": container with ID starting with 933c61e1d77c5c3d7b112435f828d16379c5e7e219797a482cc5c57783722be3 not found: ID does not exist" containerID="933c61e1d77c5c3d7b112435f828d16379c5e7e219797a482cc5c57783722be3"
Nov 02 01:57:20.911057 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.910993    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933c61e1d77c5c3d7b112435f828d16379c5e7e219797a482cc5c57783722be3"} err="failed to get container status \"933c61e1d77c5c3d7b112435f828d16379c5e7e219797a482cc5c57783722be3\": rpc error: code = NotFound desc = could not find container \"933c61e1d77c5c3d7b112435f828d16379c5e7e219797a482cc5c57783722be3\": container with ID starting with 933c61e1d77c5c3d7b112435f828d16379c5e7e219797a482cc5c57783722be3 not found: ID does not exist"
Nov 02 01:57:20.911057 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.911018    5639 scope.go:122] "RemoveContainer" containerID="9dc87c280ab43af3fd72d7b2532fcbc0505d9fc0addfd1a3bab0e42853db8e3a"
Nov 02 01:57:20.922102 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.922083    5639 scope.go:122] "RemoveContainer" containerID="9dc87c280ab43af3fd72d7b2532fcbc0505d9fc0addfd1a3bab0e42853db8e3a"
Nov 02 01:57:20.922434 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:57:20.922414    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dc87c280ab43af3fd72d7b2532fcbc0505d9fc0addfd1a3bab0e42853db8e3a\": container with ID starting with 9dc87c280ab43af3fd72d7b2532fcbc0505d9fc0addfd1a3bab0e42853db8e3a not found: ID does not exist" containerID="9dc87c280ab43af3fd72d7b2532fcbc0505d9fc0addfd1a3bab0e42853db8e3a"
Nov 02 01:57:20.922491 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:20.922448    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc87c280ab43af3fd72d7b2532fcbc0505d9fc0addfd1a3bab0e42853db8e3a"} err="failed to get container status \"9dc87c280ab43af3fd72d7b2532fcbc0505d9fc0addfd1a3bab0e42853db8e3a\": rpc error: code = NotFound desc = could not find container \"9dc87c280ab43af3fd72d7b2532fcbc0505d9fc0addfd1a3bab0e42853db8e3a\": container with ID starting with 9dc87c280ab43af3fd72d7b2532fcbc0505d9fc0addfd1a3bab0e42853db8e3a not found: ID does not exist"
Nov 02 01:57:21.064537 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:21.064508    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"]
Nov 02 01:57:21.067770 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:21.067747    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-lxdwx"]
Nov 02 01:57:21.173711 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:21.173633    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"]
Nov 02 01:57:21.180436 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:21.180410    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-5pc8r"]
Nov 02 01:57:21.225917 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:21.225885    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e35e13f-b6eb-4a4c-8142-f54a5c621a97" path="/var/lib/kubelet/pods/4e35e13f-b6eb-4a4c-8142-f54a5c621a97/volumes"
Nov 02 01:57:21.226526 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:21.226510    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd05aad1-3b5f-43bd-988e-fce23728a36c" path="/var/lib/kubelet/pods/cd05aad1-3b5f-43bd-988e-fce23728a36c/volumes"
Nov 02 01:57:27.421906 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:57:27.421858    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:57:36.064450 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:36.064412    5639 generic.go:356] "Generic (PLEG): container finished" podID="124b9d97-7aec-44dd-a24b-06be952004fc" containerID="a599f47014a914bad2b2f716f29a0bf30b66f348c4890665f5f2ace86edea778" exitCode=0
Nov 02 01:57:36.064919 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:36.064481    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd" event={"ID":"124b9d97-7aec-44dd-a24b-06be952004fc","Type":"ContainerDied","Data":"a599f47014a914bad2b2f716f29a0bf30b66f348c4890665f5f2ace86edea778"}
Nov 02 01:57:36.064919 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:36.064889    5639 scope.go:122] "RemoveContainer" containerID="a599f47014a914bad2b2f716f29a0bf30b66f348c4890665f5f2ace86edea778"
Nov 02 01:57:36.066599 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:36.066571    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:57:36.066670 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:36.066660    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:57:36.066707 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:36.066694    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:57:37.092504 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:37.092464    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd" event={"ID":"124b9d97-7aec-44dd-a24b-06be952004fc","Type":"ContainerStarted","Data":"f35931641094e1695c3108bb58b7c40445d8c2ffafa56c4180fe6d6280b14bd7"}
Nov 02 01:57:37.650856 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:37.650818    5639 scope.go:122] "RemoveContainer" containerID="16d9321f8e6b42fcd33c92c0be4a017e049f75d3dd760ac703b1f2326a1cbb0e"
Nov 02 01:57:37.684525 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:37.684480    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:57:37.684752 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:37.684731    5639 patch_prober.go:29] interesting pod/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 02 01:57:37.684841 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:37.684783    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd" podUID="124b9d97-7aec-44dd-a24b-06be952004fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 02 01:57:37.685842 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:37.685822    5639 scope.go:122] "RemoveContainer" containerID="7e3be3ffe8c7b79bf2c1ae1798d1e45ae6c9685172b3724bebb0c03fe4afbfd6"
Nov 02 01:57:37.696455 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:37.696434    5639 scope.go:122] "RemoveContainer" containerID="b9f62d5475436886ec53834c5e6ec9f9255400d3ea6c17c71fb071298efe928d"
Nov 02 01:57:37.705836 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:37.705820    5639 scope.go:122] "RemoveContainer" containerID="97b7b40d3a72ccf1b6a8c6d2019a6ed0e4a7d34d103e00ee06691209b3b2637b"
Nov 02 01:57:37.715045 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:37.715022    5639 scope.go:122] "RemoveContainer" containerID="2999e7ee14e40487b8464e2827e4cfc186bfaf4ba38e0a5f7c882819ac5a667c"
Nov 02 01:57:38.683305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:38.682703    5639 patch_prober.go:29] interesting pod/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 02 01:57:38.683305 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:38.682766    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd" podUID="124b9d97-7aec-44dd-a24b-06be952004fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 02 01:57:39.129926 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.129897    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x"]
Nov 02 01:57:39.130662 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.130647    5639 cpu_manager.go:414] "containerMap: removing container" podUID="cd05aad1-3b5f-43bd-988e-fce23728a36c" containerName="istio-proxy"
Nov 02 01:57:39.130727 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.130664    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd05aad1-3b5f-43bd-988e-fce23728a36c" containerName="istio-proxy"
Nov 02 01:57:39.130727 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.130674    5639 cpu_manager.go:414] "containerMap: removing container" podUID="4e35e13f-b6eb-4a4c-8142-f54a5c621a97" containerName="istio-proxy"
Nov 02 01:57:39.130727 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.130681    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e35e13f-b6eb-4a4c-8142-f54a5c621a97" containerName="istio-proxy"
Nov 02 01:57:39.130819 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.130805    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd05aad1-3b5f-43bd-988e-fce23728a36c" containerName="istio-proxy"
Nov 02 01:57:39.130819 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.130813    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e35e13f-b6eb-4a4c-8142-f54a5c621a97" containerName="istio-proxy"
Nov 02 01:57:39.130884 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.130821    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="86b56f9b-4c16-412c-b170-76133ab7abb2" containerName="istio-proxy"
Nov 02 01:57:39.134427 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.134403    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x"
Nov 02 01:57:39.156760 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.156730    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f21df8d-e07e-46d4-9186-408789552242-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x\" (UID: \"6f21df8d-e07e-46d4-9186-408789552242\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x"
Nov 02 01:57:39.156922 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.156837    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zscsv\" (UniqueName: \"kubernetes.io/projected/6f21df8d-e07e-46d4-9186-408789552242-kube-api-access-zscsv\") pod \"payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x\" (UID: \"6f21df8d-e07e-46d4-9186-408789552242\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x"
Nov 02 01:57:39.193214 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.193160    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x"]
Nov 02 01:57:39.205836 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.205804    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-39ceb2-dockercfg-mzsc7\""
Nov 02 01:57:39.206172 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.206152    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-39ceb2\""
Nov 02 01:57:39.400402 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.397143    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zscsv\" (UniqueName: \"kubernetes.io/projected/6f21df8d-e07e-46d4-9186-408789552242-kube-api-access-zscsv\") pod \"payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x\" (UID: \"6f21df8d-e07e-46d4-9186-408789552242\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x"
Nov 02 01:57:39.401417 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.400886    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f21df8d-e07e-46d4-9186-408789552242-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x\" (UID: \"6f21df8d-e07e-46d4-9186-408789552242\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x"
Nov 02 01:57:39.403353 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.403323    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f21df8d-e07e-46d4-9186-408789552242-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x\" (UID: \"6f21df8d-e07e-46d4-9186-408789552242\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x"
Nov 02 01:57:39.495474 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.495432    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zscsv\" (UniqueName: \"kubernetes.io/projected/6f21df8d-e07e-46d4-9186-408789552242-kube-api-access-zscsv\") pod \"payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x\" (UID: \"6f21df8d-e07e-46d4-9186-408789552242\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x"
Nov 02 01:57:39.502241 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.502206    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc"]
Nov 02 01:57:39.507068 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.507043    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc"
Nov 02 01:57:39.507833 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.507802    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc"]
Nov 02 01:57:39.606937 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.606901    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znxcx\" (UniqueName: \"kubernetes.io/projected/b1edf1cd-ee67-4762-9093-3e41dbd9df4c-kube-api-access-znxcx\") pod \"payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc\" (UID: \"b1edf1cd-ee67-4762-9093-3e41dbd9df4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc"
Nov 02 01:57:39.607119 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.607078    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1edf1cd-ee67-4762-9093-3e41dbd9df4c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc\" (UID: \"b1edf1cd-ee67-4762-9093-3e41dbd9df4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc"
Nov 02 01:57:39.681363 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.679728    5639 patch_prober.go:29] interesting pod/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 02 01:57:39.681363 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.679789    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd" podUID="124b9d97-7aec-44dd-a24b-06be952004fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 02 01:57:39.708601 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.708566    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1edf1cd-ee67-4762-9093-3e41dbd9df4c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc\" (UID: \"b1edf1cd-ee67-4762-9093-3e41dbd9df4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc"
Nov 02 01:57:39.708983 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.708721    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znxcx\" (UniqueName: \"kubernetes.io/projected/b1edf1cd-ee67-4762-9093-3e41dbd9df4c-kube-api-access-znxcx\") pod \"payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc\" (UID: \"b1edf1cd-ee67-4762-9093-3e41dbd9df4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc"
Nov 02 01:57:39.709267 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.709246    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1edf1cd-ee67-4762-9093-3e41dbd9df4c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc\" (UID: \"b1edf1cd-ee67-4762-9093-3e41dbd9df4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc"
Nov 02 01:57:39.748654 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.748625    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x"
Nov 02 01:57:39.789706 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.789362    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znxcx\" (UniqueName: \"kubernetes.io/projected/b1edf1cd-ee67-4762-9093-3e41dbd9df4c-kube-api-access-znxcx\") pod \"payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc\" (UID: \"b1edf1cd-ee67-4762-9093-3e41dbd9df4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc"
Nov 02 01:57:39.825884 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:39.825851    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc"
Nov 02 01:57:40.159770 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:57:40.159731    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f21df8d_e07e_46d4_9186_408789552242.slice/crio-ff441400d704c278a81999a094155dbf2abde9d0875b31592c2af6cb4c8c01b0 WatchSource:0}: Error finding container ff441400d704c278a81999a094155dbf2abde9d0875b31592c2af6cb4c8c01b0: Status 404 returned error can't find the container with id ff441400d704c278a81999a094155dbf2abde9d0875b31592c2af6cb4c8c01b0
Nov 02 01:57:40.187346 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:40.187315    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x"]
Nov 02 01:57:40.289194 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:57:40.289150    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1edf1cd_ee67_4762_9093_3e41dbd9df4c.slice/crio-0ea1d138954af04ed8f9d4a787f38b613fca45884873589f9a3e8240726e3c68 WatchSource:0}: Error finding container 0ea1d138954af04ed8f9d4a787f38b613fca45884873589f9a3e8240726e3c68: Status 404 returned error can't find the container with id 0ea1d138954af04ed8f9d4a787f38b613fca45884873589f9a3e8240726e3c68
Nov 02 01:57:40.289408 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:40.289385    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc"]
Nov 02 01:57:40.681306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:40.679701    5639 patch_prober.go:29] interesting pod/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 02 01:57:40.681306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:40.679761    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd" podUID="124b9d97-7aec-44dd-a24b-06be952004fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 02 01:57:41.156253 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:41.156208    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc" event={"ID":"b1edf1cd-ee67-4762-9093-3e41dbd9df4c","Type":"ContainerStarted","Data":"67504323135cb0acb763b72d7f06d48baa127f9560e87724108e85e809cdc68d"}
Nov 02 01:57:41.156757 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:41.156295    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc" event={"ID":"b1edf1cd-ee67-4762-9093-3e41dbd9df4c","Type":"ContainerStarted","Data":"0ea1d138954af04ed8f9d4a787f38b613fca45884873589f9a3e8240726e3c68"}
Nov 02 01:57:41.156757 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:41.156508    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc"
Nov 02 01:57:41.158517 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:41.158488    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x" event={"ID":"6f21df8d-e07e-46d4-9186-408789552242","Type":"ContainerStarted","Data":"6377b8a4ef4cba6ea7c5525d67f9a926258599dd595eb0bb33d3317f2b51976e"}
Nov 02 01:57:41.158648 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:41.158553    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x" event={"ID":"6f21df8d-e07e-46d4-9186-408789552242","Type":"ContainerStarted","Data":"ff441400d704c278a81999a094155dbf2abde9d0875b31592c2af6cb4c8c01b0"}
Nov 02 01:57:41.158648 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:41.158582    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x"
Nov 02 01:57:41.288006 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:41.287942    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc" podStartSLOduration=2.287920516 podStartE2EDuration="2.287920516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:57:41.273180162 +0000 UTC m=+1926.176862275" watchObservedRunningTime="2026-11-02 01:57:41.287920516 +0000 UTC m=+1926.191602630"
Nov 02 01:57:41.414386 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:41.413805    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x" podStartSLOduration=2.413782415 podStartE2EDuration="2.413782415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:57:41.389002257 +0000 UTC m=+1926.292684371" watchObservedRunningTime="2026-11-02 01:57:41.413782415 +0000 UTC m=+1926.317464530"
Nov 02 01:57:41.688223 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:41.687608    5639 patch_prober.go:29] interesting pod/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 02 01:57:41.688223 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:41.687692    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd" podUID="124b9d97-7aec-44dd-a24b-06be952004fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 02 01:57:42.679372 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:42.679338    5639 patch_prober.go:29] interesting pod/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 02 01:57:42.679784 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:42.679392    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd" podUID="124b9d97-7aec-44dd-a24b-06be952004fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 02 01:57:43.325404 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:43.325376    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:57:43.684211 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:43.684174    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:57:44.205829 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:44.205801    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 01:57:52.193249 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:52.193219    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x"
Nov 02 01:57:52.193688 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:57:52.193306    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc"
Nov 02 01:57:57.421389 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:57:57.421346    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:58:00.041803 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.041766    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"]
Nov 02 01:58:00.046259 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.046234    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.098727 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.098685    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-ef5a72-openshift-default-dockercfg-sllrl\""
Nov 02 01:58:00.153673 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.153629    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-credential-socket\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.153937 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.153918    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-envoy\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.154092 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.154060    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-workload-socket\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.154206 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.154169    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-workload-certs\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.154346 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.154274    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-data\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.154462 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.154360    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85c9d8c5-bc73-4b32-b543-adc3d9509702-istiod-ca-cert\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.154462 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.154405    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-podinfo\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.154581 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.154511    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-token\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.154581 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.154570    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt6t9\" (UniqueName: \"kubernetes.io/projected/85c9d8c5-bc73-4b32-b543-adc3d9509702-kube-api-access-vt6t9\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.155364 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.155341    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"]
Nov 02 01:58:00.262690 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.262641    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-token\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.262876 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.262742    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt6t9\" (UniqueName: \"kubernetes.io/projected/85c9d8c5-bc73-4b32-b543-adc3d9509702-kube-api-access-vt6t9\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.262973 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.262952    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-credential-socket\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.263068 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.262987    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-envoy\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.263068 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.263051    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-workload-socket\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.266558 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.263245    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-workload-certs\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.266558 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.263496    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-data\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.266558 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.263567    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85c9d8c5-bc73-4b32-b543-adc3d9509702-istiod-ca-cert\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.266558 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.263630    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-podinfo\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.266558 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.266474    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-credential-socket\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.270799 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.270745    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-data\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.271531 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.271471    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85c9d8c5-bc73-4b32-b543-adc3d9509702-istiod-ca-cert\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.271531 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.271505    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-workload-socket\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.272007 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.271784    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-workload-certs\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.275470 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.275371    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-podinfo\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.280539 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.280508    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-envoy\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.326433 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.326360    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-token\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.330852 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.330820    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt6t9\" (UniqueName: \"kubernetes.io/projected/85c9d8c5-bc73-4b32-b543-adc3d9509702-kube-api-access-vt6t9\") pod \"e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") " pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.362748 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.362710    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:00.857818 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.857770    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"]
Nov 02 01:58:00.893839 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:58:00.893809    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c9d8c5_bc73_4b32_b543_adc3d9509702.slice/crio-12fee098277d3b847ac2d7e96636d211429fb9209bceddf194a6ee10a6b80a38 WatchSource:0}: Error finding container 12fee098277d3b847ac2d7e96636d211429fb9209bceddf194a6ee10a6b80a38: Status 404 returned error can't find the container with id 12fee098277d3b847ac2d7e96636d211429fb9209bceddf194a6ee10a6b80a38
Nov 02 01:58:00.897023 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.896980    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:58:00.897103 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.897083    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:58:00.897151 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:00.897136    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:58:01.374100 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:01.374056    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276" event={"ID":"85c9d8c5-bc73-4b32-b543-adc3d9509702","Type":"ContainerStarted","Data":"7307ad5782c6bf3c83a8d5bdf5d74dceb401a2a1c965c4c0615949c242c1e3f6"}
Nov 02 01:58:01.374679 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:01.374129    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276" event={"ID":"85c9d8c5-bc73-4b32-b543-adc3d9509702","Type":"ContainerStarted","Data":"12fee098277d3b847ac2d7e96636d211429fb9209bceddf194a6ee10a6b80a38"}
Nov 02 01:58:01.771010 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:01.770890    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276" podStartSLOduration=2.770869368 podStartE2EDuration="2.770869368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:01.506971327 +0000 UTC m=+1946.410653442" watchObservedRunningTime="2026-11-02 01:58:01.770869368 +0000 UTC m=+1946.674551483"
Nov 02 01:58:01.775973 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:01.775937    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:58:02.372198 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:02.372155    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:02.387968 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:02.387934    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:02.390872 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:02.390257    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:02.392030 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:02.392003    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 01:58:12.469602 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:12.469566    5639 generic.go:356] "Generic (PLEG): container finished" podID="5d218ddb-88c3-4646-a31b-cdd5d05dbc1d" containerID="c7546833bfb3c3832e8221acfb033a02a8ff85aca3e16399304f8f04bad8afc0" exitCode=0
Nov 02 01:58:12.470049 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:12.469639    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z" event={"ID":"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d","Type":"ContainerDied","Data":"c7546833bfb3c3832e8221acfb033a02a8ff85aca3e16399304f8f04bad8afc0"}
Nov 02 01:58:12.470156 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:12.470136    5639 scope.go:122] "RemoveContainer" containerID="c7546833bfb3c3832e8221acfb033a02a8ff85aca3e16399304f8f04bad8afc0"
Nov 02 01:58:12.471994 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:12.471964    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:58:12.472080 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:12.472036    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:58:12.472080 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:12.472072    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:58:13.495370 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:13.495329    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z" event={"ID":"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d","Type":"ContainerStarted","Data":"9b967f9b1fb4766f6dfc5d32191bc725ee918927b0a17b8a49dead9b257cf89a"}
Nov 02 01:58:13.600910 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:13.600876    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:58:13.602364 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:13.602313    5639 patch_prober.go:29] interesting pod/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Nov 02 01:58:13.602364 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:13.602357    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z" podUID="5d218ddb-88c3-4646-a31b-cdd5d05dbc1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Nov 02 01:58:14.601007 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:14.600967    5639 patch_prober.go:29] interesting pod/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Nov 02 01:58:14.601476 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:14.601022    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z" podUID="5d218ddb-88c3-4646-a31b-cdd5d05dbc1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Nov 02 01:58:14.981365 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:14.980406    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:58:15.601407 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:15.601371    5639 patch_prober.go:29] interesting pod/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Nov 02 01:58:15.601823 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:15.601431    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z" podUID="5d218ddb-88c3-4646-a31b-cdd5d05dbc1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Nov 02 01:58:16.600993 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:16.600955    5639 patch_prober.go:29] interesting pod/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Nov 02 01:58:16.608317 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:16.601265    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z" podUID="5d218ddb-88c3-4646-a31b-cdd5d05dbc1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Nov 02 01:58:16.808364 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:16.808330    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"]
Nov 02 01:58:16.813072 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:16.813050    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:16.876613 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:16.875778    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c600fc-openshift-default-dockercfg-dgwzj\""
Nov 02 01:58:16.891408 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:16.889299    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-podinfo\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:16.891408 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:16.889375    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-credential-socket\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:16.891408 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:16.889459    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-envoy\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:16.891408 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:16.889499    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-token\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:16.891408 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:16.889562    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-workload-socket\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:16.891408 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:16.889620    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-workload-certs\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:16.891408 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:16.889712    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istiod-ca-cert\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:16.891408 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:16.889799    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjln4\" (UniqueName: \"kubernetes.io/projected/25cd82fe-a02f-45cf-9a41-bdbaea642b30-kube-api-access-xjln4\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:16.891408 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:16.889851    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-data\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:16.925501 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:16.924808    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"]
Nov 02 01:58:17.020303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:16.995575    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-workload-certs\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:17.020303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:16.995695    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istiod-ca-cert\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:17.020303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:16.995790    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjln4\" (UniqueName: \"kubernetes.io/projected/25cd82fe-a02f-45cf-9a41-bdbaea642b30-kube-api-access-xjln4\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:17.020303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:16.995857    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-data\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:17.020303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:16.995949    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-podinfo\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:17.020303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:16.996215    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-credential-socket\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:17.020303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:16.996326    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-envoy\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:17.020303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:16.996369    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-token\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:17.020303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:16.996441    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-workload-socket\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:17.034020 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:17.032745    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-workload-socket\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:17.034020 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:17.033913    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istiod-ca-cert\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:17.062794 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:17.062707    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-podinfo\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:17.088508 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:17.074613    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-credential-socket\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:17.124414 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:17.109448    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-data\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:17.153350 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:17.151790    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjln4\" (UniqueName: \"kubernetes.io/projected/25cd82fe-a02f-45cf-9a41-bdbaea642b30-kube-api-access-xjln4\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:17.289085 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:17.287037    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-workload-certs\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:17.303484 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:17.298902    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-envoy\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:17.321317 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:17.319899    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-token\") pod \"e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") " pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:17.429816 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:17.429727    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:17.601783 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:17.601070    5639 patch_prober.go:29] interesting pod/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Nov 02 01:58:17.601783 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:17.601153    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z" podUID="5d218ddb-88c3-4646-a31b-cdd5d05dbc1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Nov 02 01:58:17.913804 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:17.913765    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"]
Nov 02 01:58:17.915906 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:58:17.915879    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25cd82fe_a02f_45cf_9a41_bdbaea642b30.slice/crio-fe8534225f0a4e93b09a0249c5774baa6be3ae163d15732a75024d63c4f85b89 WatchSource:0}: Error finding container fe8534225f0a4e93b09a0249c5774baa6be3ae163d15732a75024d63c4f85b89: Status 404 returned error can't find the container with id fe8534225f0a4e93b09a0249c5774baa6be3ae163d15732a75024d63c4f85b89
Nov 02 01:58:17.919137 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:17.919100    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:58:17.919248 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:17.919210    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:58:17.919360 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:17.919253    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:58:18.602973 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:18.601359    5639 patch_prober.go:29] interesting pod/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Nov 02 01:58:18.602973 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:18.601421    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z" podUID="5d218ddb-88c3-4646-a31b-cdd5d05dbc1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Nov 02 01:58:18.612869 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:18.612833    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw" event={"ID":"25cd82fe-a02f-45cf-9a41-bdbaea642b30","Type":"ContainerStarted","Data":"e4a8674d8a142bb9b0d93ad7b91bb3bf25b3066656b2892728e2b62aa4a9ca6f"}
Nov 02 01:58:18.613057 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:18.612889    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw" event={"ID":"25cd82fe-a02f-45cf-9a41-bdbaea642b30","Type":"ContainerStarted","Data":"fe8534225f0a4e93b09a0249c5774baa6be3ae163d15732a75024d63c4f85b89"}
Nov 02 01:58:18.741730 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:18.741665    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw" podStartSLOduration=2.741645417 podStartE2EDuration="2.741645417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:18.735848162 +0000 UTC m=+1963.639530277" watchObservedRunningTime="2026-11-02 01:58:18.741645417 +0000 UTC m=+1963.645327530"
Nov 02 01:58:19.430824 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:19.430792    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:19.436961 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:19.436938    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:19.606449 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:19.606423    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:58:19.607720 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:19.607694    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 01:58:19.619676 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:19.619650    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:19.620454 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:19.620437    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 01:58:20.668731 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:20.668697    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz"]
Nov 02 01:58:20.673380 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:20.673356    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz"
Nov 02 01:58:20.727302 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:20.726484    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-19436b-dockercfg-fkg9w\""
Nov 02 01:58:20.727302 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:20.726497    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-19436b\""
Nov 02 01:58:20.731319 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:20.731272    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz"]
Nov 02 01:58:20.826332 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:20.826268    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpf8h\" (UniqueName: \"kubernetes.io/projected/111a0595-8aab-4dff-acc4-37b21384f047-kube-api-access-bpf8h\") pod \"payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz\" (UID: \"111a0595-8aab-4dff-acc4-37b21384f047\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz"
Nov 02 01:58:20.852525 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:20.852464    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111a0595-8aab-4dff-acc4-37b21384f047-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz\" (UID: \"111a0595-8aab-4dff-acc4-37b21384f047\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz"
Nov 02 01:58:21.006342 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:21.006188    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpf8h\" (UniqueName: \"kubernetes.io/projected/111a0595-8aab-4dff-acc4-37b21384f047-kube-api-access-bpf8h\") pod \"payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz\" (UID: \"111a0595-8aab-4dff-acc4-37b21384f047\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz"
Nov 02 01:58:21.006342 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:21.006257    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111a0595-8aab-4dff-acc4-37b21384f047-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz\" (UID: \"111a0595-8aab-4dff-acc4-37b21384f047\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz"
Nov 02 01:58:21.007332 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:21.007257    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111a0595-8aab-4dff-acc4-37b21384f047-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz\" (UID: \"111a0595-8aab-4dff-acc4-37b21384f047\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz"
Nov 02 01:58:21.025064 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:21.025034    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv"]
Nov 02 01:58:21.036576 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:21.036546    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv"]
Nov 02 01:58:21.036898 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:21.036878    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv"
Nov 02 01:58:21.069160 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:21.069116    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpf8h\" (UniqueName: \"kubernetes.io/projected/111a0595-8aab-4dff-acc4-37b21384f047-kube-api-access-bpf8h\") pod \"payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz\" (UID: \"111a0595-8aab-4dff-acc4-37b21384f047\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz"
Nov 02 01:58:21.209258 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:21.209200    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5fl6\" (UniqueName: \"kubernetes.io/projected/6487b7f6-974c-4efb-8acb-37e65e050562-kube-api-access-t5fl6\") pod \"payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv\" (UID: \"6487b7f6-974c-4efb-8acb-37e65e050562\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv"
Nov 02 01:58:21.209499 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:21.209415    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6487b7f6-974c-4efb-8acb-37e65e050562-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv\" (UID: \"6487b7f6-974c-4efb-8acb-37e65e050562\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv"
Nov 02 01:58:21.289832 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:21.289731    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz"
Nov 02 01:58:21.310944 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:21.310905    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6487b7f6-974c-4efb-8acb-37e65e050562-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv\" (UID: \"6487b7f6-974c-4efb-8acb-37e65e050562\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv"
Nov 02 01:58:21.311230 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:21.311209    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5fl6\" (UniqueName: \"kubernetes.io/projected/6487b7f6-974c-4efb-8acb-37e65e050562-kube-api-access-t5fl6\") pod \"payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv\" (UID: \"6487b7f6-974c-4efb-8acb-37e65e050562\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv"
Nov 02 01:58:21.312114 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:21.312067    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6487b7f6-974c-4efb-8acb-37e65e050562-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv\" (UID: \"6487b7f6-974c-4efb-8acb-37e65e050562\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv"
Nov 02 01:58:21.373599 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:21.373534    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5fl6\" (UniqueName: \"kubernetes.io/projected/6487b7f6-974c-4efb-8acb-37e65e050562-kube-api-access-t5fl6\") pod \"payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv\" (UID: \"6487b7f6-974c-4efb-8acb-37e65e050562\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv"
Nov 02 01:58:21.661725 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:21.661695    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv"
Nov 02 01:58:21.677240 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:21.677211    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz"]
Nov 02 01:58:21.677816 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:58:21.677566    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111a0595_8aab_4dff_acc4_37b21384f047.slice/crio-d5c9f7def9c665615134ac99ff0588999def1e3bde998ca5f22ac04fd45d87f1 WatchSource:0}: Error finding container d5c9f7def9c665615134ac99ff0588999def1e3bde998ca5f22ac04fd45d87f1: Status 404 returned error can't find the container with id d5c9f7def9c665615134ac99ff0588999def1e3bde998ca5f22ac04fd45d87f1
Nov 02 01:58:22.102949 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:58:22.102913    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6487b7f6_974c_4efb_8acb_37e65e050562.slice/crio-4f2d319f83a1949ad0f5dcdc6484826814766b5c2564e127c4cc688a41744a47 WatchSource:0}: Error finding container 4f2d319f83a1949ad0f5dcdc6484826814766b5c2564e127c4cc688a41744a47: Status 404 returned error can't find the container with id 4f2d319f83a1949ad0f5dcdc6484826814766b5c2564e127c4cc688a41744a47
Nov 02 01:58:22.104167 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:22.104126    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv"]
Nov 02 01:58:22.662797 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:22.662759    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv" event={"ID":"6487b7f6-974c-4efb-8acb-37e65e050562","Type":"ContainerStarted","Data":"f9cd24805ae21a14be05b3dc4764b3dfdeb02745eab02e613c823368ee5e6fb5"}
Nov 02 01:58:22.663036 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:22.662816    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv" event={"ID":"6487b7f6-974c-4efb-8acb-37e65e050562","Type":"ContainerStarted","Data":"4f2d319f83a1949ad0f5dcdc6484826814766b5c2564e127c4cc688a41744a47"}
Nov 02 01:58:22.663036 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:22.662857    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv"
Nov 02 01:58:22.664713 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:22.664686    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz" event={"ID":"111a0595-8aab-4dff-acc4-37b21384f047","Type":"ContainerStarted","Data":"9c121bb823e19f68ef64404df64f57491037bfa0120b5dcb2f0921ea3e4ae6b5"}
Nov 02 01:58:22.664844 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:22.664735    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz" event={"ID":"111a0595-8aab-4dff-acc4-37b21384f047","Type":"ContainerStarted","Data":"d5c9f7def9c665615134ac99ff0588999def1e3bde998ca5f22ac04fd45d87f1"}
Nov 02 01:58:22.664844 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:22.664797    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz"
Nov 02 01:58:22.783130 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:22.783055    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv" podStartSLOduration=2.783033687 podStartE2EDuration="2.783033687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:22.779735511 +0000 UTC m=+1967.683417627" watchObservedRunningTime="2026-11-02 01:58:22.783033687 +0000 UTC m=+1967.686715829"
Nov 02 01:58:22.938370 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:22.937377    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz" podStartSLOduration=2.9373547970000002 podStartE2EDuration="2.937354797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:22.917620854 +0000 UTC m=+1967.821302969" watchObservedRunningTime="2026-11-02 01:58:22.937354797 +0000 UTC m=+1967.841036912"
Nov 02 01:58:27.422091 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:58:27.422054    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:58:33.675363 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:33.675335    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz"
Nov 02 01:58:33.675852 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:33.675401    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv"
Nov 02 01:58:37.080035 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:37.080003    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm"]
Nov 02 01:58:37.084553 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:37.084532    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm"
Nov 02 01:58:37.124243 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:37.124216    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm"]
Nov 02 01:58:37.136546 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:37.136516    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-109b07\""
Nov 02 01:58:37.169406 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:37.169370    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpkj7\" (UniqueName: \"kubernetes.io/projected/b0b1ff83-fc50-4730-a4ab-e566d6c04d30-kube-api-access-wpkj7\") pod \"payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm\" (UID: \"b0b1ff83-fc50-4730-a4ab-e566d6c04d30\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm"
Nov 02 01:58:37.169550 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:37.169484    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0b1ff83-fc50-4730-a4ab-e566d6c04d30-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm\" (UID: \"b0b1ff83-fc50-4730-a4ab-e566d6c04d30\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm"
Nov 02 01:58:37.274311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:37.271501    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpkj7\" (UniqueName: \"kubernetes.io/projected/b0b1ff83-fc50-4730-a4ab-e566d6c04d30-kube-api-access-wpkj7\") pod \"payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm\" (UID: \"b0b1ff83-fc50-4730-a4ab-e566d6c04d30\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm"
Nov 02 01:58:37.274311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:37.271571    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0b1ff83-fc50-4730-a4ab-e566d6c04d30-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm\" (UID: \"b0b1ff83-fc50-4730-a4ab-e566d6c04d30\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm"
Nov 02 01:58:37.274311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:37.272721    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0b1ff83-fc50-4730-a4ab-e566d6c04d30-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm\" (UID: \"b0b1ff83-fc50-4730-a4ab-e566d6c04d30\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm"
Nov 02 01:58:37.362360 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:37.362052    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpkj7\" (UniqueName: \"kubernetes.io/projected/b0b1ff83-fc50-4730-a4ab-e566d6c04d30-kube-api-access-wpkj7\") pod \"payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm\" (UID: \"b0b1ff83-fc50-4730-a4ab-e566d6c04d30\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm"
Nov 02 01:58:37.405302 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:37.403140    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm"
Nov 02 01:58:37.435607 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:37.435568    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd"]
Nov 02 01:58:37.440614 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:37.440580    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd"]
Nov 02 01:58:37.440750 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:37.440702    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd"
Nov 02 01:58:37.474677 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:37.474351    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mbfn\" (UniqueName: \"kubernetes.io/projected/ff588a60-de17-4af0-9710-c8ec8fad7ef6-kube-api-access-4mbfn\") pod \"payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd\" (UID: \"ff588a60-de17-4af0-9710-c8ec8fad7ef6\") " pod="openshift-ingress/payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd"
Nov 02 01:58:37.474677 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:37.474483    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff588a60-de17-4af0-9710-c8ec8fad7ef6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd\" (UID: \"ff588a60-de17-4af0-9710-c8ec8fad7ef6\") " pod="openshift-ingress/payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd"
Nov 02 01:58:37.493662 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:37.493633    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-109b07-dockercfg-b6dgx\""
Nov 02 01:58:37.576868 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:37.576832    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mbfn\" (UniqueName: \"kubernetes.io/projected/ff588a60-de17-4af0-9710-c8ec8fad7ef6-kube-api-access-4mbfn\") pod \"payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd\" (UID: \"ff588a60-de17-4af0-9710-c8ec8fad7ef6\") " pod="openshift-ingress/payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd"
Nov 02 01:58:37.577050 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:37.576933    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff588a60-de17-4af0-9710-c8ec8fad7ef6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd\" (UID: \"ff588a60-de17-4af0-9710-c8ec8fad7ef6\") " pod="openshift-ingress/payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd"
Nov 02 01:58:37.577625 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:37.577606    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff588a60-de17-4af0-9710-c8ec8fad7ef6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd\" (UID: \"ff588a60-de17-4af0-9710-c8ec8fad7ef6\") " pod="openshift-ingress/payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd"
Nov 02 01:58:37.638086 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:37.638011    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mbfn\" (UniqueName: \"kubernetes.io/projected/ff588a60-de17-4af0-9710-c8ec8fad7ef6-kube-api-access-4mbfn\") pod \"payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd\" (UID: \"ff588a60-de17-4af0-9710-c8ec8fad7ef6\") " pod="openshift-ingress/payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd"
Nov 02 01:58:37.774028 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:37.773991    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd"
Nov 02 01:58:37.814921 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:37.814888    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm"]
Nov 02 01:58:37.822791 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:58:37.822755    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b1ff83_fc50_4730_a4ab_e566d6c04d30.slice/crio-5cfbcf24be756891485ce72c5cc248fd6df25645d639adc7c37b8a91aa6d6497 WatchSource:0}: Error finding container 5cfbcf24be756891485ce72c5cc248fd6df25645d639adc7c37b8a91aa6d6497: Status 404 returned error can't find the container with id 5cfbcf24be756891485ce72c5cc248fd6df25645d639adc7c37b8a91aa6d6497
Nov 02 01:58:37.905330 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:37.902684    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm" event={"ID":"b0b1ff83-fc50-4730-a4ab-e566d6c04d30","Type":"ContainerStarted","Data":"5cfbcf24be756891485ce72c5cc248fd6df25645d639adc7c37b8a91aa6d6497"}
Nov 02 01:58:38.161217 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:38.161187    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd"]
Nov 02 01:58:38.161646 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:58:38.161236    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff588a60_de17_4af0_9710_c8ec8fad7ef6.slice/crio-65a18f7b55a26a8110bf3bf45096521a02cd467828a346b6ba4dd8d14e11bf27 WatchSource:0}: Error finding container 65a18f7b55a26a8110bf3bf45096521a02cd467828a346b6ba4dd8d14e11bf27: Status 404 returned error can't find the container with id 65a18f7b55a26a8110bf3bf45096521a02cd467828a346b6ba4dd8d14e11bf27
Nov 02 01:58:38.911686 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:38.911644    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd" event={"ID":"ff588a60-de17-4af0-9710-c8ec8fad7ef6","Type":"ContainerStarted","Data":"d90987704b4c7cfd628a0f0d2fe34c0e4a651cf6427f1bdaf88bf20455b0e441"}
Nov 02 01:58:38.911908 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:38.911721    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd"
Nov 02 01:58:38.911908 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:38.911748    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd" event={"ID":"ff588a60-de17-4af0-9710-c8ec8fad7ef6","Type":"ContainerStarted","Data":"65a18f7b55a26a8110bf3bf45096521a02cd467828a346b6ba4dd8d14e11bf27"}
Nov 02 01:58:38.913470 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:38.913444    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm" event={"ID":"b0b1ff83-fc50-4730-a4ab-e566d6c04d30","Type":"ContainerStarted","Data":"716aea416f18197d1c2451aa4b4b26687112ba9e73d214d24245c82be80055d2"}
Nov 02 01:58:38.913590 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:38.913493    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm"
Nov 02 01:58:39.031761 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:39.031703    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd" podStartSLOduration=2.031685267 podStartE2EDuration="2.031685267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:39.027858099 +0000 UTC m=+1983.931540213" watchObservedRunningTime="2026-11-02 01:58:39.031685267 +0000 UTC m=+1983.935367384"
Nov 02 01:58:39.191370 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:39.182173    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm" podStartSLOduration=2.182150703 podStartE2EDuration="2.182150703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:39.17242678 +0000 UTC m=+1984.076108927" watchObservedRunningTime="2026-11-02 01:58:39.182150703 +0000 UTC m=+1984.085832819"
Nov 02 01:58:43.050110 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:43.050075    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b1d120c-67n6k"]
Nov 02 01:58:43.099391 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:43.099350    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b1d120c-67n6k"]
Nov 02 01:58:43.223754 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:43.223715    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76863b48-cafa-4e55-adb6-f98efc604043" path="/var/lib/kubelet/pods/76863b48-cafa-4e55-adb6-f98efc604043/volumes"
Nov 02 01:58:44.871941 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:44.871906    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7051-998df6"]
Nov 02 01:58:44.877578 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:44.877557    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7051-998df6"
Nov 02 01:58:44.930680 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:44.930648    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7051-998df6"]
Nov 02 01:58:44.975745 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:44.975713    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-n9czn\""
Nov 02 01:58:45.012721 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:45.012684    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gs86\" (UniqueName: \"kubernetes.io/projected/1a73d54d-661f-4180-bd44-3a9d3cb2d976-kube-api-access-7gs86\") pod \"test-curl-7051-998df6\" (UID: \"1a73d54d-661f-4180-bd44-3a9d3cb2d976\") " pod="opendatahub/test-curl-7051-998df6"
Nov 02 01:58:45.113980 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:45.113943    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gs86\" (UniqueName: \"kubernetes.io/projected/1a73d54d-661f-4180-bd44-3a9d3cb2d976-kube-api-access-7gs86\") pod \"test-curl-7051-998df6\" (UID: \"1a73d54d-661f-4180-bd44-3a9d3cb2d976\") " pod="opendatahub/test-curl-7051-998df6"
Nov 02 01:58:45.175063 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:45.174986    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gs86\" (UniqueName: \"kubernetes.io/projected/1a73d54d-661f-4180-bd44-3a9d3cb2d976-kube-api-access-7gs86\") pod \"test-curl-7051-998df6\" (UID: \"1a73d54d-661f-4180-bd44-3a9d3cb2d976\") " pod="opendatahub/test-curl-7051-998df6"
Nov 02 01:58:45.191978 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:45.191942    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7051-998df6"
Nov 02 01:58:45.577990 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:45.577952    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7051-998df6"]
Nov 02 01:58:45.578395 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:58:45.578368    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a73d54d_661f_4180_bd44_3a9d3cb2d976.slice/crio-be72a1e32a2ba77242e39429e3bb9b7347e8d34bafa5b3da844cf05c565a1d12 WatchSource:0}: Error finding container be72a1e32a2ba77242e39429e3bb9b7347e8d34bafa5b3da844cf05c565a1d12: Status 404 returned error can't find the container with id be72a1e32a2ba77242e39429e3bb9b7347e8d34bafa5b3da844cf05c565a1d12
Nov 02 01:58:46.028888 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:46.028852    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7051-998df6" event={"ID":"1a73d54d-661f-4180-bd44-3a9d3cb2d976","Type":"ContainerStarted","Data":"4c9bd8b7c8657eaa6f90b35f04dd30fc8999dba8440ef171aeadc4559ad5f457"}
Nov 02 01:58:46.029364 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:46.028899    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7051-998df6" event={"ID":"1a73d54d-661f-4180-bd44-3a9d3cb2d976","Type":"ContainerStarted","Data":"be72a1e32a2ba77242e39429e3bb9b7347e8d34bafa5b3da844cf05c565a1d12"}
Nov 02 01:58:46.143508 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:46.143447    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7051-998df6" podStartSLOduration=2.143432545 podStartE2EDuration="2.143432545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:46.140754338 +0000 UTC m=+1991.044436454" watchObservedRunningTime="2026-11-02 01:58:46.143432545 +0000 UTC m=+1991.047114658"
Nov 02 01:58:47.652019 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:47.651740    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7051-998df6"]
Nov 02 01:58:47.656533 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:47.656499    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7051-998df6"]
Nov 02 01:58:48.046485 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:48.046378    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7051-998df6" podUID="1a73d54d-661f-4180-bd44-3a9d3cb2d976" containerName="test-curl-7051-998df6" containerID="cri-o://4c9bd8b7c8657eaa6f90b35f04dd30fc8999dba8440ef171aeadc4559ad5f457" gracePeriod=2
Nov 02 01:58:48.097719 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:58:48.097679    5639 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7051-998df6\" is forbidden: User \"system:node:ip-10-0-136-53.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-53.us-west-1.compute.internal' and this object" podUID="1a73d54d-661f-4180-bd44-3a9d3cb2d976" pod="opendatahub/test-curl-7051-998df6"
Nov 02 01:58:48.682714 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:48.682676    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk"]
Nov 02 01:58:48.683727 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:48.683702    5639 cpu_manager.go:414] "containerMap: removing container" podUID="1a73d54d-661f-4180-bd44-3a9d3cb2d976" containerName="test-curl-7051-998df6"
Nov 02 01:58:48.683727 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:48.683727    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a73d54d-661f-4180-bd44-3a9d3cb2d976" containerName="test-curl-7051-998df6"
Nov 02 01:58:48.683949 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:48.683935    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a73d54d-661f-4180-bd44-3a9d3cb2d976" containerName="test-curl-7051-998df6"
Nov 02 01:58:48.687907 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:48.687885    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk"
Nov 02 01:58:48.739598 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:58:48.739430    5639 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7051-998df6\" is forbidden: User \"system:node:ip-10-0-136-53.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-53.us-west-1.compute.internal' and this object" podUID="1a73d54d-661f-4180-bd44-3a9d3cb2d976" pod="opendatahub/test-curl-7051-998df6"
Nov 02 01:58:48.740296 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:48.740235    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:58:48.740296 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:48.740249    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:58:48.744211 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:48.744187    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk"]
Nov 02 01:58:48.789160 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:48.789127    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:58:48.789356 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:48.789259    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:58:48.880619 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:48.880587    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b605a17f-6063-48fd-bc46-19de1a0ed9ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk\" (UID: \"b605a17f-6063-48fd-bc46-19de1a0ed9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk"
Nov 02 01:58:48.880820 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:48.880802    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b605a17f-6063-48fd-bc46-19de1a0ed9ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk\" (UID: \"b605a17f-6063-48fd-bc46-19de1a0ed9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk"
Nov 02 01:58:48.880882 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:48.880865    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6pf2\" (UniqueName: \"kubernetes.io/projected/b605a17f-6063-48fd-bc46-19de1a0ed9ef-kube-api-access-z6pf2\") pod \"maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk\" (UID: \"b605a17f-6063-48fd-bc46-19de1a0ed9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk"
Nov 02 01:58:48.982804 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:48.982718    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6pf2\" (UniqueName: \"kubernetes.io/projected/b605a17f-6063-48fd-bc46-19de1a0ed9ef-kube-api-access-z6pf2\") pod \"maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk\" (UID: \"b605a17f-6063-48fd-bc46-19de1a0ed9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk"
Nov 02 01:58:48.982960 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:48.982856    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b605a17f-6063-48fd-bc46-19de1a0ed9ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk\" (UID: \"b605a17f-6063-48fd-bc46-19de1a0ed9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk"
Nov 02 01:58:48.983050 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:48.983031    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b605a17f-6063-48fd-bc46-19de1a0ed9ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk\" (UID: \"b605a17f-6063-48fd-bc46-19de1a0ed9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk"
Nov 02 01:58:48.983691 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:48.983667    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b605a17f-6063-48fd-bc46-19de1a0ed9ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk\" (UID: \"b605a17f-6063-48fd-bc46-19de1a0ed9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk"
Nov 02 01:58:48.985385 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:48.985363    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b605a17f-6063-48fd-bc46-19de1a0ed9ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk\" (UID: \"b605a17f-6063-48fd-bc46-19de1a0ed9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk"
Nov 02 01:58:49.043941 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:49.043914    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6pf2\" (UniqueName: \"kubernetes.io/projected/b605a17f-6063-48fd-bc46-19de1a0ed9ef-kube-api-access-z6pf2\") pod \"maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk\" (UID: \"b605a17f-6063-48fd-bc46-19de1a0ed9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk"
Nov 02 01:58:49.301978 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:49.301902    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk"
Nov 02 01:58:49.683180 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:49.683150    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk"]
Nov 02 01:58:49.684347 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:58:49.684320    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb605a17f_6063_48fd_bc46_19de1a0ed9ef.slice/crio-a13640bdbc49cdc8802ddd5525e6849f01c503d0c280aaade3ebe72eba5a5e8e WatchSource:0}: Error finding container a13640bdbc49cdc8802ddd5525e6849f01c503d0c280aaade3ebe72eba5a5e8e: Status 404 returned error can't find the container with id a13640bdbc49cdc8802ddd5525e6849f01c503d0c280aaade3ebe72eba5a5e8e
Nov 02 01:58:49.926694 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:49.926665    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm"
Nov 02 01:58:49.926878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:49.926750    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd"
Nov 02 01:58:50.079142 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:50.078965    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk" event={"ID":"b605a17f-6063-48fd-bc46-19de1a0ed9ef","Type":"ContainerStarted","Data":"a13640bdbc49cdc8802ddd5525e6849f01c503d0c280aaade3ebe72eba5a5e8e"}
Nov 02 01:58:50.082424 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:50.082387    5639 generic.go:356] "Generic (PLEG): container finished" podID="1a73d54d-661f-4180-bd44-3a9d3cb2d976" containerID="4c9bd8b7c8657eaa6f90b35f04dd30fc8999dba8440ef171aeadc4559ad5f457" exitCode=137
Nov 02 01:58:50.309961 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:50.309898    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7051-998df6"
Nov 02 01:58:50.405687 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:50.405656    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a73d54d-661f-4180-bd44-3a9d3cb2d976-kube-api-access-7gs86\" (UniqueName: \"kubernetes.io/projected/1a73d54d-661f-4180-bd44-3a9d3cb2d976-kube-api-access-7gs86\") pod \"1a73d54d-661f-4180-bd44-3a9d3cb2d976\" (UID: \"1a73d54d-661f-4180-bd44-3a9d3cb2d976\") "
Nov 02 01:58:50.407917 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:50.407892    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a73d54d-661f-4180-bd44-3a9d3cb2d976-kube-api-access-7gs86" pod "1a73d54d-661f-4180-bd44-3a9d3cb2d976" (UID: "1a73d54d-661f-4180-bd44-3a9d3cb2d976"). InnerVolumeSpecName "kube-api-access-7gs86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:58:50.507208 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:50.507180    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gs86\" (UniqueName: \"kubernetes.io/projected/1a73d54d-661f-4180-bd44-3a9d3cb2d976-kube-api-access-7gs86\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:58:51.090852 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:51.090808    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7051-998df6"
Nov 02 01:58:51.090852 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:51.090845    5639 scope.go:122] "RemoveContainer" containerID="4c9bd8b7c8657eaa6f90b35f04dd30fc8999dba8440ef171aeadc4559ad5f457"
Nov 02 01:58:51.093103 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:51.093077    5639 generic.go:356] "Generic (PLEG): container finished" podID="b605a17f-6063-48fd-bc46-19de1a0ed9ef" containerID="75ac57b658a4da3c5dd35512022e6788a47d8c62fe7878c64ccc42eafe3540ed" exitCode=0
Nov 02 01:58:51.093232 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:51.093164    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk" event={"ID":"b605a17f-6063-48fd-bc46-19de1a0ed9ef","Type":"ContainerDied","Data":"75ac57b658a4da3c5dd35512022e6788a47d8c62fe7878c64ccc42eafe3540ed"}
Nov 02 01:58:51.221504 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:51.221477    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a73d54d-661f-4180-bd44-3a9d3cb2d976" path="/var/lib/kubelet/pods/1a73d54d-661f-4180-bd44-3a9d3cb2d976/volumes"
Nov 02 01:58:51.259383 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:58:51.259306    5639 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7051-998df6\" is forbidden: User \"system:node:ip-10-0-136-53.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-53.us-west-1.compute.internal' and this object" podUID="1a73d54d-661f-4180-bd44-3a9d3cb2d976" pod="opendatahub/test-curl-7051-998df6"
Nov 02 01:58:52.288728 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:52.288705    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk"
Nov 02 01:58:52.358380 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:52.347970    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b605a17f-6063-48fd-bc46-19de1a0ed9ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b605a17f-6063-48fd-bc46-19de1a0ed9ef-maas-api-xks-ca\") pod \"b605a17f-6063-48fd-bc46-19de1a0ed9ef\" (UID: \"b605a17f-6063-48fd-bc46-19de1a0ed9ef\") "
Nov 02 01:58:52.358380 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:52.348171    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b605a17f-6063-48fd-bc46-19de1a0ed9ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b605a17f-6063-48fd-bc46-19de1a0ed9ef-maas-api-openshift-ca\") pod \"b605a17f-6063-48fd-bc46-19de1a0ed9ef\" (UID: \"b605a17f-6063-48fd-bc46-19de1a0ed9ef\") "
Nov 02 01:58:52.358380 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:52.348536    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b605a17f-6063-48fd-bc46-19de1a0ed9ef-kube-api-access-z6pf2\" (UniqueName: \"kubernetes.io/projected/b605a17f-6063-48fd-bc46-19de1a0ed9ef-kube-api-access-z6pf2\") pod \"b605a17f-6063-48fd-bc46-19de1a0ed9ef\" (UID: \"b605a17f-6063-48fd-bc46-19de1a0ed9ef\") "
Nov 02 01:58:52.358380 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:52.356480    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b605a17f-6063-48fd-bc46-19de1a0ed9ef-maas-api-openshift-ca" pod "b605a17f-6063-48fd-bc46-19de1a0ed9ef" (UID: "b605a17f-6063-48fd-bc46-19de1a0ed9ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:58:52.385458 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:52.381218    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b605a17f-6063-48fd-bc46-19de1a0ed9ef-kube-api-access-z6pf2" pod "b605a17f-6063-48fd-bc46-19de1a0ed9ef" (UID: "b605a17f-6063-48fd-bc46-19de1a0ed9ef"). InnerVolumeSpecName "kube-api-access-z6pf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:58:52.403600 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:52.403545    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b605a17f-6063-48fd-bc46-19de1a0ed9ef-maas-api-xks-ca" pod "b605a17f-6063-48fd-bc46-19de1a0ed9ef" (UID: "b605a17f-6063-48fd-bc46-19de1a0ed9ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:58:52.466329 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:52.454994    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6pf2\" (UniqueName: \"kubernetes.io/projected/b605a17f-6063-48fd-bc46-19de1a0ed9ef-kube-api-access-z6pf2\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:58:52.466329 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:52.455030    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b605a17f-6063-48fd-bc46-19de1a0ed9ef-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:58:52.466329 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:52.455048    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b605a17f-6063-48fd-bc46-19de1a0ed9ef-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:58:52.561382 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:52.561349    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 01:58:53.111311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:53.111260    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk"
Nov 02 01:58:53.111311 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:53.111273    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk" event={"ID":"b605a17f-6063-48fd-bc46-19de1a0ed9ef","Type":"ContainerDied","Data":"a13640bdbc49cdc8802ddd5525e6849f01c503d0c280aaade3ebe72eba5a5e8e"}
Nov 02 01:58:53.111574 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:53.111362    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a13640bdbc49cdc8802ddd5525e6849f01c503d0c280aaade3ebe72eba5a5e8e"
Nov 02 01:58:57.436351 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:58:57.436312    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:58:59.050141 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:59.050112    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3a72816-tcd6q"]
Nov 02 01:58:59.111392 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:59.111352    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3a72816-tcd6q"]
Nov 02 01:58:59.223082 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:58:59.223042    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b04800a-455b-497e-b3a9-7ee75f8e62f6" path="/var/lib/kubelet/pods/3b04800a-455b-497e-b3a9-7ee75f8e62f6/volumes"
Nov 02 01:59:00.174550 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:00.174467    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg"]
Nov 02 01:59:00.288993 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:00.288961    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw"]
Nov 02 01:59:00.528571 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:00.528492    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"]
Nov 02 01:59:00.528747 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:00.528708    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp" podUID="02d298db-5bc9-469a-af6b-ba7fa45f8602" containerName="istio-proxy" containerID="cri-o://3d0d09d742168662cec0cff79ef1c1312b95607a52508c79d5b443c99cf1110d" gracePeriod=30
Nov 02 01:59:02.013688 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:02.013328    5639 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:59:02.013688 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:02.013407    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp" podUID="02d298db-5bc9-469a-af6b-ba7fa45f8602" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:59:05.175843 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.175760    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg" podUID="6424735d-18ec-462d-bb38-104055e1e001" containerName="payload-processing" containerID="cri-o://66e07aaf5f8d155fc75336b313986f5ea311d5278e422134f7811c54de25e445" gracePeriod=25
Nov 02 01:59:05.282899 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.282867    5639 generic.go:356] "Generic (PLEG): container finished" podID="b2e9988f-011a-4a80-ad65-d54e5266e8dd" containerID="ed62dd24cfa8cc54208107256e01590ea5f2a6782a2a17b58ee356598ee85378" exitCode=0
Nov 02 01:59:05.283081 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.282952    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf" event={"ID":"b2e9988f-011a-4a80-ad65-d54e5266e8dd","Type":"ContainerDied","Data":"ed62dd24cfa8cc54208107256e01590ea5f2a6782a2a17b58ee356598ee85378"}
Nov 02 01:59:05.284127 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.284099    5639 scope.go:122] "RemoveContainer" containerID="ed62dd24cfa8cc54208107256e01590ea5f2a6782a2a17b58ee356598ee85378"
Nov 02 01:59:05.285830 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.285803    5639 generic.go:356] "Generic (PLEG): container finished" podID="5d8da016-d523-4efd-91de-aae563f7f1a6" containerID="325a54703dad3b6d5437e231b5ef8bcdc8902b51ca8435ee5ee39c9f432b6603" exitCode=0
Nov 02 01:59:05.286415 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.286064    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx" event={"ID":"5d8da016-d523-4efd-91de-aae563f7f1a6","Type":"ContainerDied","Data":"325a54703dad3b6d5437e231b5ef8bcdc8902b51ca8435ee5ee39c9f432b6603"}
Nov 02 01:59:05.287395 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.287372    5639 scope.go:122] "RemoveContainer" containerID="325a54703dad3b6d5437e231b5ef8bcdc8902b51ca8435ee5ee39c9f432b6603"
Nov 02 01:59:05.290502 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.288858    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:59:05.290502 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.289024    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:59:05.290502 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.289268    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:59:05.290870 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.290845    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw" podUID="7157f0f1-b793-41ee-ab6a-cdf56fd7d72c" containerName="payload-pre-processing" containerID="cri-o://4bdd010ad62c5fad0216c6d3eb9c3f0b15eddbdbee8dcbf8424ebc23bc73b1c6" gracePeriod=25
Nov 02 01:59:05.303495 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.297468    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:59:05.303495 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.297635    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:59:05.303495 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.297736    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 01:59:05.494026 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.492638    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg"
Nov 02 01:59:05.648567 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.641731    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6424735d-18ec-462d-bb38-104055e1e001-kube-api-access-dgrkz\" (UniqueName: \"kubernetes.io/projected/6424735d-18ec-462d-bb38-104055e1e001-kube-api-access-dgrkz\") pod \"6424735d-18ec-462d-bb38-104055e1e001\" (UID: \"6424735d-18ec-462d-bb38-104055e1e001\") "
Nov 02 01:59:05.648567 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.642194    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6424735d-18ec-462d-bb38-104055e1e001-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6424735d-18ec-462d-bb38-104055e1e001-plugins-config-volume\") pod \"6424735d-18ec-462d-bb38-104055e1e001\" (UID: \"6424735d-18ec-462d-bb38-104055e1e001\") "
Nov 02 01:59:05.648567 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.643618    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6424735d-18ec-462d-bb38-104055e1e001-plugins-config-volume" pod "6424735d-18ec-462d-bb38-104055e1e001" (UID: "6424735d-18ec-462d-bb38-104055e1e001"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:59:05.650156 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.650093    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6424735d-18ec-462d-bb38-104055e1e001-kube-api-access-dgrkz" pod "6424735d-18ec-462d-bb38-104055e1e001" (UID: "6424735d-18ec-462d-bb38-104055e1e001"). InnerVolumeSpecName "kube-api-access-dgrkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:59:05.694764 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.685117    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw"
Nov 02 01:59:05.749505 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.748293    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7157f0f1-b793-41ee-ab6a-cdf56fd7d72c-kube-api-access-nk6hp\" (UniqueName: \"kubernetes.io/projected/7157f0f1-b793-41ee-ab6a-cdf56fd7d72c-kube-api-access-nk6hp\") pod \"7157f0f1-b793-41ee-ab6a-cdf56fd7d72c\" (UID: \"7157f0f1-b793-41ee-ab6a-cdf56fd7d72c\") "
Nov 02 01:59:05.749505 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.749009    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgrkz\" (UniqueName: \"kubernetes.io/projected/6424735d-18ec-462d-bb38-104055e1e001-kube-api-access-dgrkz\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:05.749505 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.749036    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6424735d-18ec-462d-bb38-104055e1e001-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:05.854536 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.854052    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7157f0f1-b793-41ee-ab6a-cdf56fd7d72c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7157f0f1-b793-41ee-ab6a-cdf56fd7d72c-plugins-config-volume\") pod \"7157f0f1-b793-41ee-ab6a-cdf56fd7d72c\" (UID: \"7157f0f1-b793-41ee-ab6a-cdf56fd7d72c\") "
Nov 02 01:59:05.946833 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.946739    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7157f0f1-b793-41ee-ab6a-cdf56fd7d72c-plugins-config-volume" pod "7157f0f1-b793-41ee-ab6a-cdf56fd7d72c" (UID: "7157f0f1-b793-41ee-ab6a-cdf56fd7d72c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:59:05.960449 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.957463    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7157f0f1-b793-41ee-ab6a-cdf56fd7d72c-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:05.960449 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:05.959491    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7157f0f1-b793-41ee-ab6a-cdf56fd7d72c-kube-api-access-nk6hp" pod "7157f0f1-b793-41ee-ab6a-cdf56fd7d72c" (UID: "7157f0f1-b793-41ee-ab6a-cdf56fd7d72c"). InnerVolumeSpecName "kube-api-access-nk6hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:59:06.062927 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.062149    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk6hp\" (UniqueName: \"kubernetes.io/projected/7157f0f1-b793-41ee-ab6a-cdf56fd7d72c-kube-api-access-nk6hp\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:06.201077 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.201005    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:59:06.295232 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.295195    5639 generic.go:356] "Generic (PLEG): container finished" podID="6424735d-18ec-462d-bb38-104055e1e001" containerID="66e07aaf5f8d155fc75336b313986f5ea311d5278e422134f7811c54de25e445" exitCode=0
Nov 02 01:59:06.295468 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.295245    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg"
Nov 02 01:59:06.295468 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.295321    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg" event={"ID":"6424735d-18ec-462d-bb38-104055e1e001","Type":"ContainerDied","Data":"66e07aaf5f8d155fc75336b313986f5ea311d5278e422134f7811c54de25e445"}
Nov 02 01:59:06.295468 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.295373    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg" event={"ID":"6424735d-18ec-462d-bb38-104055e1e001","Type":"ContainerDied","Data":"fd65ea0ad42f8aab07de612e5d07f2aa4adfbb7316c33d17ede8190a7f8cf817"}
Nov 02 01:59:06.295468 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.295398    5639 scope.go:122] "RemoveContainer" containerID="66e07aaf5f8d155fc75336b313986f5ea311d5278e422134f7811c54de25e445"
Nov 02 01:59:06.297011 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.296796    5639 generic.go:356] "Generic (PLEG): container finished" podID="02d298db-5bc9-469a-af6b-ba7fa45f8602" containerID="3d0d09d742168662cec0cff79ef1c1312b95607a52508c79d5b443c99cf1110d" exitCode=0
Nov 02 01:59:06.297011 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.296897    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp" event={"ID":"02d298db-5bc9-469a-af6b-ba7fa45f8602","Type":"ContainerDied","Data":"3d0d09d742168662cec0cff79ef1c1312b95607a52508c79d5b443c99cf1110d"}
Nov 02 01:59:06.297011 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.296934    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp" event={"ID":"02d298db-5bc9-469a-af6b-ba7fa45f8602","Type":"ContainerDied","Data":"31bc616ea18fa5033ff6b6487ebee038ad63b4ab326586722ebe34e1320bafcf"}
Nov 02 01:59:06.297011 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.296999    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"
Nov 02 01:59:06.299611 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.299584    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx" event={"ID":"5d8da016-d523-4efd-91de-aae563f7f1a6","Type":"ContainerStarted","Data":"4bbc475bcfb487be3b93e77093d30391682727e237959b009b40910ad2bd6f4a"}
Nov 02 01:59:06.301254 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.301231    5639 generic.go:356] "Generic (PLEG): container finished" podID="7157f0f1-b793-41ee-ab6a-cdf56fd7d72c" containerID="4bdd010ad62c5fad0216c6d3eb9c3f0b15eddbdbee8dcbf8424ebc23bc73b1c6" exitCode=0
Nov 02 01:59:06.301415 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.301271    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw"
Nov 02 01:59:06.301415 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.301344    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw" event={"ID":"7157f0f1-b793-41ee-ab6a-cdf56fd7d72c","Type":"ContainerDied","Data":"4bdd010ad62c5fad0216c6d3eb9c3f0b15eddbdbee8dcbf8424ebc23bc73b1c6"}
Nov 02 01:59:06.301518 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.301439    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw" event={"ID":"7157f0f1-b793-41ee-ab6a-cdf56fd7d72c","Type":"ContainerDied","Data":"0b3e3359f9225ec21766ea44a22ca05cbefae443b54785863baf583e695f7c39"}
Nov 02 01:59:06.303664 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.303645    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf" event={"ID":"b2e9988f-011a-4a80-ad65-d54e5266e8dd","Type":"ContainerStarted","Data":"397a4da7f345c8732a14e16debefca5d88a4f39c3cbc66991ce02c624224df3d"}
Nov 02 01:59:06.310293 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.310251    5639 scope.go:122] "RemoveContainer" containerID="66e07aaf5f8d155fc75336b313986f5ea311d5278e422134f7811c54de25e445"
Nov 02 01:59:06.310674 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:59:06.310647    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66e07aaf5f8d155fc75336b313986f5ea311d5278e422134f7811c54de25e445\": container with ID starting with 66e07aaf5f8d155fc75336b313986f5ea311d5278e422134f7811c54de25e445 not found: ID does not exist" containerID="66e07aaf5f8d155fc75336b313986f5ea311d5278e422134f7811c54de25e445"
Nov 02 01:59:06.310761 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.310685    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66e07aaf5f8d155fc75336b313986f5ea311d5278e422134f7811c54de25e445"} err="failed to get container status \"66e07aaf5f8d155fc75336b313986f5ea311d5278e422134f7811c54de25e445\": rpc error: code = NotFound desc = could not find container \"66e07aaf5f8d155fc75336b313986f5ea311d5278e422134f7811c54de25e445\": container with ID starting with 66e07aaf5f8d155fc75336b313986f5ea311d5278e422134f7811c54de25e445 not found: ID does not exist"
Nov 02 01:59:06.310761 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.310707    5639 scope.go:122] "RemoveContainer" containerID="3d0d09d742168662cec0cff79ef1c1312b95607a52508c79d5b443c99cf1110d"
Nov 02 01:59:06.321324 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.321302    5639 scope.go:122] "RemoveContainer" containerID="3d0d09d742168662cec0cff79ef1c1312b95607a52508c79d5b443c99cf1110d"
Nov 02 01:59:06.321591 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:59:06.321573    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d0d09d742168662cec0cff79ef1c1312b95607a52508c79d5b443c99cf1110d\": container with ID starting with 3d0d09d742168662cec0cff79ef1c1312b95607a52508c79d5b443c99cf1110d not found: ID does not exist" containerID="3d0d09d742168662cec0cff79ef1c1312b95607a52508c79d5b443c99cf1110d"
Nov 02 01:59:06.321637 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.321598    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d0d09d742168662cec0cff79ef1c1312b95607a52508c79d5b443c99cf1110d"} err="failed to get container status \"3d0d09d742168662cec0cff79ef1c1312b95607a52508c79d5b443c99cf1110d\": rpc error: code = NotFound desc = could not find container \"3d0d09d742168662cec0cff79ef1c1312b95607a52508c79d5b443c99cf1110d\": container with ID starting with 3d0d09d742168662cec0cff79ef1c1312b95607a52508c79d5b443c99cf1110d not found: ID does not exist"
Nov 02 01:59:06.321637 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.321614    5639 scope.go:122] "RemoveContainer" containerID="4bdd010ad62c5fad0216c6d3eb9c3f0b15eddbdbee8dcbf8424ebc23bc73b1c6"
Nov 02 01:59:06.335163 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.335142    5639 scope.go:122] "RemoveContainer" containerID="4bdd010ad62c5fad0216c6d3eb9c3f0b15eddbdbee8dcbf8424ebc23bc73b1c6"
Nov 02 01:59:06.335544 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:59:06.335522    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bdd010ad62c5fad0216c6d3eb9c3f0b15eddbdbee8dcbf8424ebc23bc73b1c6\": container with ID starting with 4bdd010ad62c5fad0216c6d3eb9c3f0b15eddbdbee8dcbf8424ebc23bc73b1c6 not found: ID does not exist" containerID="4bdd010ad62c5fad0216c6d3eb9c3f0b15eddbdbee8dcbf8424ebc23bc73b1c6"
Nov 02 01:59:06.335629 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.335557    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bdd010ad62c5fad0216c6d3eb9c3f0b15eddbdbee8dcbf8424ebc23bc73b1c6"} err="failed to get container status \"4bdd010ad62c5fad0216c6d3eb9c3f0b15eddbdbee8dcbf8424ebc23bc73b1c6\": rpc error: code = NotFound desc = could not find container \"4bdd010ad62c5fad0216c6d3eb9c3f0b15eddbdbee8dcbf8424ebc23bc73b1c6\": container with ID starting with 4bdd010ad62c5fad0216c6d3eb9c3f0b15eddbdbee8dcbf8424ebc23bc73b1c6 not found: ID does not exist"
Nov 02 01:59:06.366057 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.366015    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-podinfo\") pod \"02d298db-5bc9-469a-af6b-ba7fa45f8602\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") "
Nov 02 01:59:06.366258 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.366105    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-data\") pod \"02d298db-5bc9-469a-af6b-ba7fa45f8602\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") "
Nov 02 01:59:06.366258 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.366165    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-token\" (UniqueName: \"kubernetes.io/projected/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-token\") pod \"02d298db-5bc9-469a-af6b-ba7fa45f8602\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") "
Nov 02 01:59:06.366258 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.366230    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-workload-socket\") pod \"02d298db-5bc9-469a-af6b-ba7fa45f8602\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") "
Nov 02 01:59:06.366533 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.366266    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-credential-socket\") pod \"02d298db-5bc9-469a-af6b-ba7fa45f8602\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") "
Nov 02 01:59:06.366533 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.366371    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02d298db-5bc9-469a-af6b-ba7fa45f8602-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d298db-5bc9-469a-af6b-ba7fa45f8602-istiod-ca-cert\") pod \"02d298db-5bc9-469a-af6b-ba7fa45f8602\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") "
Nov 02 01:59:06.366533 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.366437    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-workload-certs\") pod \"02d298db-5bc9-469a-af6b-ba7fa45f8602\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") "
Nov 02 01:59:06.366533 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.366468    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-envoy\") pod \"02d298db-5bc9-469a-af6b-ba7fa45f8602\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") "
Nov 02 01:59:06.366533 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.366495    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-data" pod "02d298db-5bc9-469a-af6b-ba7fa45f8602" (UID: "02d298db-5bc9-469a-af6b-ba7fa45f8602"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:59:06.366814 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.366640    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-credential-socket" pod "02d298db-5bc9-469a-af6b-ba7fa45f8602" (UID: "02d298db-5bc9-469a-af6b-ba7fa45f8602"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:59:06.366814 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.366663    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02d298db-5bc9-469a-af6b-ba7fa45f8602-kube-api-access-pw42f\" (UniqueName: \"kubernetes.io/projected/02d298db-5bc9-469a-af6b-ba7fa45f8602-kube-api-access-pw42f\") pod \"02d298db-5bc9-469a-af6b-ba7fa45f8602\" (UID: \"02d298db-5bc9-469a-af6b-ba7fa45f8602\") "
Nov 02 01:59:06.366922 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.366843    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-workload-socket" pod "02d298db-5bc9-469a-af6b-ba7fa45f8602" (UID: "02d298db-5bc9-469a-af6b-ba7fa45f8602"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:59:06.366922 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.366884    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02d298db-5bc9-469a-af6b-ba7fa45f8602-istiod-ca-cert" pod "02d298db-5bc9-469a-af6b-ba7fa45f8602" (UID: "02d298db-5bc9-469a-af6b-ba7fa45f8602"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:59:06.367065 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.367036    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-workload-certs" pod "02d298db-5bc9-469a-af6b-ba7fa45f8602" (UID: "02d298db-5bc9-469a-af6b-ba7fa45f8602"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:59:06.367732 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.367709    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:06.367878 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.367863    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:06.367984 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.367972    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:06.368074 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.368062    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:06.368163 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.368150    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d298db-5bc9-469a-af6b-ba7fa45f8602-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:06.369145 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.369095    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-token" pod "02d298db-5bc9-469a-af6b-ba7fa45f8602" (UID: "02d298db-5bc9-469a-af6b-ba7fa45f8602"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:59:06.370475 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.370449    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-envoy" pod "02d298db-5bc9-469a-af6b-ba7fa45f8602" (UID: "02d298db-5bc9-469a-af6b-ba7fa45f8602"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:59:06.370563 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.370499    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-podinfo" pod "02d298db-5bc9-469a-af6b-ba7fa45f8602" (UID: "02d298db-5bc9-469a-af6b-ba7fa45f8602"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:59:06.370616 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.370562    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d298db-5bc9-469a-af6b-ba7fa45f8602-kube-api-access-pw42f" pod "02d298db-5bc9-469a-af6b-ba7fa45f8602" (UID: "02d298db-5bc9-469a-af6b-ba7fa45f8602"). InnerVolumeSpecName "kube-api-access-pw42f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:59:06.476368 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.473798    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:06.476368 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.473837    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw42f\" (UniqueName: \"kubernetes.io/projected/02d298db-5bc9-469a-af6b-ba7fa45f8602-kube-api-access-pw42f\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:06.476368 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.473852    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:06.476368 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.473865    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d298db-5bc9-469a-af6b-ba7fa45f8602-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:06.509854 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.509343    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw"]
Nov 02 01:59:06.513175 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.512476    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-6tjgw"]
Nov 02 01:59:06.614378 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.614343    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg"]
Nov 02 01:59:06.618480 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.618447    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-kd5bg"]
Nov 02 01:59:06.630291 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.630253    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:59:06.631524 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.631502    5639 patch_prober.go:29] interesting pod/e2e-api-a-109b07-openshift-default-54786bf495-brglf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 02 01:59:06.631649 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.631544    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf" podUID="b2e9988f-011a-4a80-ad65-d54e5266e8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 02 01:59:06.642867 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.642840    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:59:06.643898 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.643875    5639 patch_prober.go:29] interesting pod/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Nov 02 01:59:06.643998 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:06.643923    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx" podUID="5d8da016-d523-4efd-91de-aae563f7f1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Nov 02 01:59:07.110479 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:07.110440    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"]
Nov 02 01:59:07.114466 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:07.114437    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-pcrsp"]
Nov 02 01:59:07.222997 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:07.222967    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02d298db-5bc9-469a-af6b-ba7fa45f8602" path="/var/lib/kubelet/pods/02d298db-5bc9-469a-af6b-ba7fa45f8602/volumes"
Nov 02 01:59:07.223534 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:07.223516    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6424735d-18ec-462d-bb38-104055e1e001" path="/var/lib/kubelet/pods/6424735d-18ec-462d-bb38-104055e1e001/volumes"
Nov 02 01:59:07.223924 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:07.223911    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7157f0f1-b793-41ee-ab6a-cdf56fd7d72c" path="/var/lib/kubelet/pods/7157f0f1-b793-41ee-ab6a-cdf56fd7d72c/volumes"
Nov 02 01:59:07.630614 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:07.630581    5639 patch_prober.go:29] interesting pod/e2e-api-a-109b07-openshift-default-54786bf495-brglf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 02 01:59:07.630830 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:07.630634    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf" podUID="b2e9988f-011a-4a80-ad65-d54e5266e8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 02 01:59:07.643128 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:07.643099    5639 patch_prober.go:29] interesting pod/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Nov 02 01:59:07.643208 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:07.643152    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx" podUID="5d8da016-d523-4efd-91de-aae563f7f1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Nov 02 01:59:07.790323 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:07.790073    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:59:08.630546 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:08.630503    5639 patch_prober.go:29] interesting pod/e2e-api-a-109b07-openshift-default-54786bf495-brglf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 02 01:59:08.630948 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:08.630574    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf" podUID="b2e9988f-011a-4a80-ad65-d54e5266e8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 02 01:59:08.642647 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:08.642617    5639 patch_prober.go:29] interesting pod/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Nov 02 01:59:08.642754 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:08.642667    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx" podUID="5d8da016-d523-4efd-91de-aae563f7f1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Nov 02 01:59:08.654390 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:08.654364    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:59:09.630771 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:09.630741    5639 patch_prober.go:29] interesting pod/e2e-api-a-109b07-openshift-default-54786bf495-brglf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 02 01:59:09.631169 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:09.630800    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf" podUID="b2e9988f-011a-4a80-ad65-d54e5266e8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 02 01:59:09.642326 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:09.642251    5639 patch_prober.go:29] interesting pod/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Nov 02 01:59:09.642493 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:09.642373    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx" podUID="5d8da016-d523-4efd-91de-aae563f7f1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Nov 02 01:59:10.630770 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:10.630721    5639 patch_prober.go:29] interesting pod/e2e-api-a-109b07-openshift-default-54786bf495-brglf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 02 01:59:10.631270 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:10.631223    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf" podUID="b2e9988f-011a-4a80-ad65-d54e5266e8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 02 01:59:10.642677 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:10.642651    5639 patch_prober.go:29] interesting pod/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Nov 02 01:59:10.642847 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:10.642821    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx" podUID="5d8da016-d523-4efd-91de-aae563f7f1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Nov 02 01:59:11.630240 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:11.630197    5639 patch_prober.go:29] interesting pod/e2e-api-a-109b07-openshift-default-54786bf495-brglf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 02 01:59:11.630496 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:11.630264    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf" podUID="b2e9988f-011a-4a80-ad65-d54e5266e8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 02 01:59:11.643068 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:11.643031    5639 patch_prober.go:29] interesting pod/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Nov 02 01:59:11.643496 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:11.643088    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx" podUID="5d8da016-d523-4efd-91de-aae563f7f1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Nov 02 01:59:12.635678 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:12.635647    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:59:12.636963 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:12.636937    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 01:59:12.646891 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:12.646860    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:59:12.648043 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:12.648019    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 01:59:27.424994 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:59:27.424949    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:59:37.819799 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:37.819767    5639 scope.go:122] "RemoveContainer" containerID="c5d78b841a6509a7bc6aedb22d11518db87b5bf98310d53e361879d4a818cf14"
Nov 02 01:59:39.174387 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.166795    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw"]
Nov 02 01:59:39.174387 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.167964    5639 cpu_manager.go:414] "containerMap: removing container" podUID="02d298db-5bc9-469a-af6b-ba7fa45f8602" containerName="istio-proxy"
Nov 02 01:59:39.174387 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.167987    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02d298db-5bc9-469a-af6b-ba7fa45f8602" containerName="istio-proxy"
Nov 02 01:59:39.174387 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.168020    5639 cpu_manager.go:414] "containerMap: removing container" podUID="6424735d-18ec-462d-bb38-104055e1e001" containerName="payload-processing"
Nov 02 01:59:39.174387 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.168029    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6424735d-18ec-462d-bb38-104055e1e001" containerName="payload-processing"
Nov 02 01:59:39.174387 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.168046    5639 cpu_manager.go:414] "containerMap: removing container" podUID="b605a17f-6063-48fd-bc46-19de1a0ed9ef" containerName="revoke-keys"
Nov 02 01:59:39.174387 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.168053    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b605a17f-6063-48fd-bc46-19de1a0ed9ef" containerName="revoke-keys"
Nov 02 01:59:39.174387 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.168131    5639 cpu_manager.go:414] "containerMap: removing container" podUID="7157f0f1-b793-41ee-ab6a-cdf56fd7d72c" containerName="payload-pre-processing"
Nov 02 01:59:39.174387 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.168139    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7157f0f1-b793-41ee-ab6a-cdf56fd7d72c" containerName="payload-pre-processing"
Nov 02 01:59:39.184431 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.184398    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="7157f0f1-b793-41ee-ab6a-cdf56fd7d72c" containerName="payload-pre-processing"
Nov 02 01:59:39.186120 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.186089    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="02d298db-5bc9-469a-af6b-ba7fa45f8602" containerName="istio-proxy"
Nov 02 01:59:39.186383 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.186368    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="6424735d-18ec-462d-bb38-104055e1e001" containerName="payload-processing"
Nov 02 01:59:39.186494 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.186483    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="b605a17f-6063-48fd-bc46-19de1a0ed9ef" containerName="revoke-keys"
Nov 02 01:59:39.202636 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.202605    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw"
Nov 02 01:59:39.330041 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.329999    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-ef5a72\""
Nov 02 01:59:39.337036 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.337009    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-ef5a72-dockercfg-lgpfh\""
Nov 02 01:59:39.379897 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.379860    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw"]
Nov 02 01:59:39.383146 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.383116    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb"]
Nov 02 01:59:39.388100 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.388066    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb"
Nov 02 01:59:39.396569 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.396533    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b20cafe-5759-403c-b4f8-ed895c71de18-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb\" (UID: \"1b20cafe-5759-403c-b4f8-ed895c71de18\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb"
Nov 02 01:59:39.397914 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.397852    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjgf6\" (UniqueName: \"kubernetes.io/projected/1b20cafe-5759-403c-b4f8-ed895c71de18-kube-api-access-qjgf6\") pod \"payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb\" (UID: \"1b20cafe-5759-403c-b4f8-ed895c71de18\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb"
Nov 02 01:59:39.398204 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.398184    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1f54d2-0cbb-42a0-96fd-1fea08daae41-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw\" (UID: \"4b1f54d2-0cbb-42a0-96fd-1fea08daae41\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw"
Nov 02 01:59:39.398489 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.398468    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xstrm\" (UniqueName: \"kubernetes.io/projected/4b1f54d2-0cbb-42a0-96fd-1fea08daae41-kube-api-access-xstrm\") pod \"payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw\" (UID: \"4b1f54d2-0cbb-42a0-96fd-1fea08daae41\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw"
Nov 02 01:59:39.404092 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.404051    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb"]
Nov 02 01:59:39.500551 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.500451    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1f54d2-0cbb-42a0-96fd-1fea08daae41-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw\" (UID: \"4b1f54d2-0cbb-42a0-96fd-1fea08daae41\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw"
Nov 02 01:59:39.500727 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.500659    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xstrm\" (UniqueName: \"kubernetes.io/projected/4b1f54d2-0cbb-42a0-96fd-1fea08daae41-kube-api-access-xstrm\") pod \"payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw\" (UID: \"4b1f54d2-0cbb-42a0-96fd-1fea08daae41\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw"
Nov 02 01:59:39.500727 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.500712    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b20cafe-5759-403c-b4f8-ed895c71de18-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb\" (UID: \"1b20cafe-5759-403c-b4f8-ed895c71de18\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb"
Nov 02 01:59:39.500850 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.500788    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjgf6\" (UniqueName: \"kubernetes.io/projected/1b20cafe-5759-403c-b4f8-ed895c71de18-kube-api-access-qjgf6\") pod \"payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb\" (UID: \"1b20cafe-5759-403c-b4f8-ed895c71de18\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb"
Nov 02 01:59:39.501202 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.501164    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1f54d2-0cbb-42a0-96fd-1fea08daae41-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw\" (UID: \"4b1f54d2-0cbb-42a0-96fd-1fea08daae41\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw"
Nov 02 01:59:39.501574 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.501553    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b20cafe-5759-403c-b4f8-ed895c71de18-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb\" (UID: \"1b20cafe-5759-403c-b4f8-ed895c71de18\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb"
Nov 02 01:59:39.561608 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.561576    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xstrm\" (UniqueName: \"kubernetes.io/projected/4b1f54d2-0cbb-42a0-96fd-1fea08daae41-kube-api-access-xstrm\") pod \"payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw\" (UID: \"4b1f54d2-0cbb-42a0-96fd-1fea08daae41\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw"
Nov 02 01:59:39.561767 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.561576    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjgf6\" (UniqueName: \"kubernetes.io/projected/1b20cafe-5759-403c-b4f8-ed895c71de18-kube-api-access-qjgf6\") pod \"payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb\" (UID: \"1b20cafe-5759-403c-b4f8-ed895c71de18\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb"
Nov 02 01:59:39.676743 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.676705    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw"
Nov 02 01:59:39.713197 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:39.713157    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb"
Nov 02 01:59:40.065232 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:40.065201    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw"]
Nov 02 01:59:40.065496 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:59:40.065476    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1f54d2_0cbb_42a0_96fd_1fea08daae41.slice/crio-4623f12c4cdfe42cb06d0262f3730e465aecc2ddfcd708793dc481751fd7afb0 WatchSource:0}: Error finding container 4623f12c4cdfe42cb06d0262f3730e465aecc2ddfcd708793dc481751fd7afb0: Status 404 returned error can't find the container with id 4623f12c4cdfe42cb06d0262f3730e465aecc2ddfcd708793dc481751fd7afb0
Nov 02 01:59:40.123527 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:40.123499    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb"]
Nov 02 01:59:40.124308 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:59:40.124253    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b20cafe_5759_403c_b4f8_ed895c71de18.slice/crio-8812b6c1186a6da16fe57aed803272b920d6cf06442c9e0bf8bae690cb96f9cb WatchSource:0}: Error finding container 8812b6c1186a6da16fe57aed803272b920d6cf06442c9e0bf8bae690cb96f9cb: Status 404 returned error can't find the container with id 8812b6c1186a6da16fe57aed803272b920d6cf06442c9e0bf8bae690cb96f9cb
Nov 02 01:59:40.647074 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:40.647034    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw" event={"ID":"4b1f54d2-0cbb-42a0-96fd-1fea08daae41","Type":"ContainerStarted","Data":"d95d5a50ddc20b0864b248f84bbbf98865c067b18539f05818471e8487097117"}
Nov 02 01:59:40.647579 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:40.647099    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw" event={"ID":"4b1f54d2-0cbb-42a0-96fd-1fea08daae41","Type":"ContainerStarted","Data":"4623f12c4cdfe42cb06d0262f3730e465aecc2ddfcd708793dc481751fd7afb0"}
Nov 02 01:59:40.647579 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:40.647130    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw"
Nov 02 01:59:40.649025 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:40.648998    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb" event={"ID":"1b20cafe-5759-403c-b4f8-ed895c71de18","Type":"ContainerStarted","Data":"71b2111c5fa7d05e012028a59bf0db0d080604cfef9eefd3cc784cf37f8f17d9"}
Nov 02 01:59:40.649154 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:40.649046    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb" event={"ID":"1b20cafe-5759-403c-b4f8-ed895c71de18","Type":"ContainerStarted","Data":"8812b6c1186a6da16fe57aed803272b920d6cf06442c9e0bf8bae690cb96f9cb"}
Nov 02 01:59:40.649154 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:40.649082    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb"
Nov 02 01:59:40.765549 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:40.765493    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw" podStartSLOduration=1.765472639 podStartE2EDuration="1.765472639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:59:40.76172204 +0000 UTC m=+2045.665404178" watchObservedRunningTime="2026-11-02 01:59:40.765472639 +0000 UTC m=+2045.669154751"
Nov 02 01:59:40.903373 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:40.889332    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb" podStartSLOduration=1.889311142 podStartE2EDuration="1.889311142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:59:40.888763349 +0000 UTC m=+2045.792445462" watchObservedRunningTime="2026-11-02 01:59:40.889311142 +0000 UTC m=+2045.792993257"
Nov 02 01:59:51.662341 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:51.662255    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb"
Nov 02 01:59:51.662852 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:51.662386    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw"
Nov 02 01:59:54.061212 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:54.061178    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96"]
Nov 02 01:59:54.064965 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:54.064941    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96"
Nov 02 01:59:54.122784 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:54.122753    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96"]
Nov 02 01:59:54.167012 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:54.166978    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:59:54.167195 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:54.167171    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 01:59:54.167393 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:54.167376    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:59:54.167454 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:54.167378    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:59:54.175766 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:54.175737    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbcceaa5-440d-4605-b0ed-7b0836ceb2bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96\" (UID: \"fbcceaa5-440d-4605-b0ed-7b0836ceb2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96"
Nov 02 01:59:54.175906 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:54.175812    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbcceaa5-440d-4605-b0ed-7b0836ceb2bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96\" (UID: \"fbcceaa5-440d-4605-b0ed-7b0836ceb2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96"
Nov 02 01:59:54.175906 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:54.175868    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx97p\" (UniqueName: \"kubernetes.io/projected/fbcceaa5-440d-4605-b0ed-7b0836ceb2bf-kube-api-access-gx97p\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96\" (UID: \"fbcceaa5-440d-4605-b0ed-7b0836ceb2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96"
Nov 02 01:59:54.277384 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:54.277341    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbcceaa5-440d-4605-b0ed-7b0836ceb2bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96\" (UID: \"fbcceaa5-440d-4605-b0ed-7b0836ceb2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96"
Nov 02 01:59:54.277384 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:54.277389    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbcceaa5-440d-4605-b0ed-7b0836ceb2bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96\" (UID: \"fbcceaa5-440d-4605-b0ed-7b0836ceb2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96"
Nov 02 01:59:54.277598 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:54.277425    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx97p\" (UniqueName: \"kubernetes.io/projected/fbcceaa5-440d-4605-b0ed-7b0836ceb2bf-kube-api-access-gx97p\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96\" (UID: \"fbcceaa5-440d-4605-b0ed-7b0836ceb2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96"
Nov 02 01:59:54.278113 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:54.278089    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbcceaa5-440d-4605-b0ed-7b0836ceb2bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96\" (UID: \"fbcceaa5-440d-4605-b0ed-7b0836ceb2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96"
Nov 02 01:59:54.279836 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:54.279808    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbcceaa5-440d-4605-b0ed-7b0836ceb2bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96\" (UID: \"fbcceaa5-440d-4605-b0ed-7b0836ceb2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96"
Nov 02 01:59:54.338098 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:54.338069    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx97p\" (UniqueName: \"kubernetes.io/projected/fbcceaa5-440d-4605-b0ed-7b0836ceb2bf-kube-api-access-gx97p\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96\" (UID: \"fbcceaa5-440d-4605-b0ed-7b0836ceb2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96"
Nov 02 01:59:54.379089 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:54.379057    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96"
Nov 02 01:59:54.761069 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:54.761033    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96"]
Nov 02 01:59:54.761209 ip-10-0-136-53 kubenswrapper[5639]: W1102 01:59:54.761161    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbcceaa5_440d_4605_b0ed_7b0836ceb2bf.slice/crio-963c4052aff013670787e0e6a46691d54bbd844fb99da5089eb6ee8a1de55b77 WatchSource:0}: Error finding container 963c4052aff013670787e0e6a46691d54bbd844fb99da5089eb6ee8a1de55b77: Status 404 returned error can't find the container with id 963c4052aff013670787e0e6a46691d54bbd844fb99da5089eb6ee8a1de55b77
Nov 02 01:59:55.803042 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:55.803004    5639 generic.go:356] "Generic (PLEG): container finished" podID="fbcceaa5-440d-4605-b0ed-7b0836ceb2bf" containerID="c6e00145db7ebfd6af23e0e7ab221f7055ead2812f5968fff1a1b1e915b77aea" exitCode=0
Nov 02 01:59:55.814302 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:55.804138    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96" event={"ID":"fbcceaa5-440d-4605-b0ed-7b0836ceb2bf","Type":"ContainerDied","Data":"c6e00145db7ebfd6af23e0e7ab221f7055ead2812f5968fff1a1b1e915b77aea"}
Nov 02 01:59:55.814641 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:55.814605    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96" event={"ID":"fbcceaa5-440d-4605-b0ed-7b0836ceb2bf","Type":"ContainerStarted","Data":"963c4052aff013670787e0e6a46691d54bbd844fb99da5089eb6ee8a1de55b77"}
Nov 02 01:59:57.003893 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:57.003866    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96"
Nov 02 01:59:57.032108 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:57.032079    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbcceaa5-440d-4605-b0ed-7b0836ceb2bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbcceaa5-440d-4605-b0ed-7b0836ceb2bf-maas-api-xks-ca\") pod \"fbcceaa5-440d-4605-b0ed-7b0836ceb2bf\" (UID: \"fbcceaa5-440d-4605-b0ed-7b0836ceb2bf\") "
Nov 02 01:59:57.032232 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:57.032188    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbcceaa5-440d-4605-b0ed-7b0836ceb2bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbcceaa5-440d-4605-b0ed-7b0836ceb2bf-maas-api-openshift-ca\") pod \"fbcceaa5-440d-4605-b0ed-7b0836ceb2bf\" (UID: \"fbcceaa5-440d-4605-b0ed-7b0836ceb2bf\") "
Nov 02 01:59:57.032346 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:57.032327    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbcceaa5-440d-4605-b0ed-7b0836ceb2bf-kube-api-access-gx97p\" (UniqueName: \"kubernetes.io/projected/fbcceaa5-440d-4605-b0ed-7b0836ceb2bf-kube-api-access-gx97p\") pod \"fbcceaa5-440d-4605-b0ed-7b0836ceb2bf\" (UID: \"fbcceaa5-440d-4605-b0ed-7b0836ceb2bf\") "
Nov 02 01:59:57.032558 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:57.032536    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbcceaa5-440d-4605-b0ed-7b0836ceb2bf-maas-api-openshift-ca" pod "fbcceaa5-440d-4605-b0ed-7b0836ceb2bf" (UID: "fbcceaa5-440d-4605-b0ed-7b0836ceb2bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:59:57.032791 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:57.032773    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbcceaa5-440d-4605-b0ed-7b0836ceb2bf-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:57.034354 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:57.034326    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbcceaa5-440d-4605-b0ed-7b0836ceb2bf-maas-api-xks-ca" pod "fbcceaa5-440d-4605-b0ed-7b0836ceb2bf" (UID: "fbcceaa5-440d-4605-b0ed-7b0836ceb2bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:59:57.034476 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:57.034460    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbcceaa5-440d-4605-b0ed-7b0836ceb2bf-kube-api-access-gx97p" pod "fbcceaa5-440d-4605-b0ed-7b0836ceb2bf" (UID: "fbcceaa5-440d-4605-b0ed-7b0836ceb2bf"). InnerVolumeSpecName "kube-api-access-gx97p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:59:57.134102 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:57.134065    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbcceaa5-440d-4605-b0ed-7b0836ceb2bf-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:57.134102 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:57.134103    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx97p\" (UniqueName: \"kubernetes.io/projected/fbcceaa5-440d-4605-b0ed-7b0836ceb2bf-kube-api-access-gx97p\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:57.452373 ip-10-0-136-53 kubenswrapper[5639]: E1102 01:59:57.452264    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 01:59:57.821874 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:57.821789    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96"
Nov 02 01:59:57.821874 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:57.821827    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96" event={"ID":"fbcceaa5-440d-4605-b0ed-7b0836ceb2bf","Type":"ContainerDied","Data":"963c4052aff013670787e0e6a46691d54bbd844fb99da5089eb6ee8a1de55b77"}
Nov 02 01:59:57.821874 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:57.821871    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="963c4052aff013670787e0e6a46691d54bbd844fb99da5089eb6ee8a1de55b77"
Nov 02 01:59:59.464302 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:59.461900    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk"]
Nov 02 01:59:59.464302 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:59.463175    5639 cpu_manager.go:414] "containerMap: removing container" podUID="fbcceaa5-440d-4605-b0ed-7b0836ceb2bf" containerName="revoke-keys"
Nov 02 01:59:59.464302 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:59.463199    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbcceaa5-440d-4605-b0ed-7b0836ceb2bf" containerName="revoke-keys"
Nov 02 01:59:59.464302 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:59.463519    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbcceaa5-440d-4605-b0ed-7b0836ceb2bf" containerName="revoke-keys"
Nov 02 01:59:59.469687 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:59.467331    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk"
Nov 02 01:59:59.504611 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:59.504576    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05e9bda3-da6b-437a-9b42-3a9a9a4aafcd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk\" (UID: \"05e9bda3-da6b-437a-9b42-3a9a9a4aafcd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk"
Nov 02 01:59:59.504881 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:59.504861    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbbvg\" (UniqueName: \"kubernetes.io/projected/05e9bda3-da6b-437a-9b42-3a9a9a4aafcd-kube-api-access-sbbvg\") pod \"payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk\" (UID: \"05e9bda3-da6b-437a-9b42-3a9a9a4aafcd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk"
Nov 02 01:59:59.523913 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:59.523883    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk"]
Nov 02 01:59:59.601303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:59.582195    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c600fc\""
Nov 02 01:59:59.601303 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:59.582403    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c600fc-dockercfg-kkcfp\""
Nov 02 01:59:59.617306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:59.606404    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbbvg\" (UniqueName: \"kubernetes.io/projected/05e9bda3-da6b-437a-9b42-3a9a9a4aafcd-kube-api-access-sbbvg\") pod \"payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk\" (UID: \"05e9bda3-da6b-437a-9b42-3a9a9a4aafcd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk"
Nov 02 01:59:59.617306 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:59.606550    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05e9bda3-da6b-437a-9b42-3a9a9a4aafcd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk\" (UID: \"05e9bda3-da6b-437a-9b42-3a9a9a4aafcd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk"
Nov 02 01:59:59.626587 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:59.620000    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05e9bda3-da6b-437a-9b42-3a9a9a4aafcd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk\" (UID: \"05e9bda3-da6b-437a-9b42-3a9a9a4aafcd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk"
Nov 02 01:59:59.740696 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:59.729353    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5"]
Nov 02 01:59:59.748003 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:59.741918    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5"]
Nov 02 01:59:59.748003 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:59.742035    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5"
Nov 02 01:59:59.757238 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:59.757200    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbbvg\" (UniqueName: \"kubernetes.io/projected/05e9bda3-da6b-437a-9b42-3a9a9a4aafcd-kube-api-access-sbbvg\") pod \"payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk\" (UID: \"05e9bda3-da6b-437a-9b42-3a9a9a4aafcd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk"
Nov 02 01:59:59.789524 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:59.789498    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk"
Nov 02 01:59:59.823441 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:59.823408    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f9e832d-76eb-41c1-91b9-db784e55c47e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5\" (UID: \"1f9e832d-76eb-41c1-91b9-db784e55c47e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5"
Nov 02 01:59:59.823630 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:59.823613    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9rww\" (UniqueName: \"kubernetes.io/projected/1f9e832d-76eb-41c1-91b9-db784e55c47e-kube-api-access-n9rww\") pod \"payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5\" (UID: \"1f9e832d-76eb-41c1-91b9-db784e55c47e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5"
Nov 02 01:59:59.924738 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:59.924704    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f9e832d-76eb-41c1-91b9-db784e55c47e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5\" (UID: \"1f9e832d-76eb-41c1-91b9-db784e55c47e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5"
Nov 02 01:59:59.924936 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:59.924866    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9rww\" (UniqueName: \"kubernetes.io/projected/1f9e832d-76eb-41c1-91b9-db784e55c47e-kube-api-access-n9rww\") pod \"payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5\" (UID: \"1f9e832d-76eb-41c1-91b9-db784e55c47e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5"
Nov 02 01:59:59.925492 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:59.925468    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f9e832d-76eb-41c1-91b9-db784e55c47e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5\" (UID: \"1f9e832d-76eb-41c1-91b9-db784e55c47e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5"
Nov 02 01:59:59.986046 ip-10-0-136-53 kubenswrapper[5639]: I1102 01:59:59.986014    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9rww\" (UniqueName: \"kubernetes.io/projected/1f9e832d-76eb-41c1-91b9-db784e55c47e-kube-api-access-n9rww\") pod \"payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5\" (UID: \"1f9e832d-76eb-41c1-91b9-db784e55c47e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5"
Nov 02 02:00:00.090976 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.090942    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5"
Nov 02 02:00:00.176396 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.176364    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc"]
Nov 02 02:00:00.205364 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.205333    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7"]
Nov 02 02:00:00.206615 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.206584    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc"
Nov 02 02:00:00.213939 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.213915    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw"]
Nov 02 02:00:00.214145 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.214127    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7"
Nov 02 02:00:00.220505 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.220483    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk"]
Nov 02 02:00:00.220768 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.220745    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps"]
Nov 02 02:00:00.221815 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.220699    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw"
Nov 02 02:00:00.224908 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.224885    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps"
Nov 02 02:00:00.228382 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.228359    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2"]
Nov 02 02:00:00.229526 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.229496    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa637e0-ccb8-4713-a16b-f4c353d3cb8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc\" (UID: \"3fa637e0-ccb8-4713-a16b-f4c353d3cb8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc"
Nov 02 02:00:00.230495 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.230470    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdv2m\" (UniqueName: \"kubernetes.io/projected/5dabb512-1926-413b-87ab-2a9c87e685f7-kube-api-access-kdv2m\") pod \"maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7\" (UID: \"5dabb512-1926-413b-87ab-2a9c87e685f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7"
Nov 02 02:00:00.230755 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:00:00.230647    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e9bda3_da6b_437a_9b42_3a9a9a4aafcd.slice/crio-93e70366ba65f96d327e92b1de4c4a1c7922c9e5e9e891f78f0b7a24a317e5b9 WatchSource:0}: Error finding container 93e70366ba65f96d327e92b1de4c4a1c7922c9e5e9e891f78f0b7a24a317e5b9: Status 404 returned error can't find the container with id 93e70366ba65f96d327e92b1de4c4a1c7922c9e5e9e891f78f0b7a24a317e5b9
Nov 02 02:00:00.231038 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.230785    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvfsp\" (UniqueName: \"kubernetes.io/projected/3fa637e0-ccb8-4713-a16b-f4c353d3cb8f-kube-api-access-xvfsp\") pod \"maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc\" (UID: \"3fa637e0-ccb8-4713-a16b-f4c353d3cb8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc"
Nov 02 02:00:00.231185 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.231123    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dabb512-1926-413b-87ab-2a9c87e685f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7\" (UID: \"5dabb512-1926-413b-87ab-2a9c87e685f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7"
Nov 02 02:00:00.233773 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.233753    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498"]
Nov 02 02:00:00.234592 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.234569    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2"
Nov 02 02:00:00.241600 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.241584    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt"]
Nov 02 02:00:00.242502 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.242480    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498"
Nov 02 02:00:00.254474 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.254452    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z"]
Nov 02 02:00:00.261434 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.261388    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt"
Nov 02 02:00:00.272572 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.269467    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9"]
Nov 02 02:00:00.272572 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.269594    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z"
Nov 02 02:00:00.273386 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.273363    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh"]
Nov 02 02:00:00.273635 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.273446    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9"
Nov 02 02:00:00.274506 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.274482    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:00:00.274616 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.274526    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:00:00.274678 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.274641    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:00:00.294221 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.294193    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc"]
Nov 02 02:00:00.294469 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.294450    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh"
Nov 02 02:00:00.300659 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.300623    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7"]
Nov 02 02:00:00.300776 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.300673    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc"]
Nov 02 02:00:00.300776 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.300702    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw"]
Nov 02 02:00:00.300776 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.300729    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm"]
Nov 02 02:00:00.301054 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.301038    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc"
Nov 02 02:00:00.305345 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.305320    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps"]
Nov 02 02:00:00.305443 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.305363    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2"]
Nov 02 02:00:00.305443 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.305390    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt"]
Nov 02 02:00:00.305443 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.305431    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm"
Nov 02 02:00:00.307183 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.307166    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498"]
Nov 02 02:00:00.310485 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.310467    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z"]
Nov 02 02:00:00.321917 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.321897    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm"]
Nov 02 02:00:00.336747 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.336712    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm\" (UID: \"3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm"
Nov 02 02:00:00.337023 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.336974    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b669073a-3244-4c17-9255-197abd78c544-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498\" (UID: \"b669073a-3244-4c17-9255-197abd78c544\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498"
Nov 02 02:00:00.337293 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.337255    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa637e0-ccb8-4713-a16b-f4c353d3cb8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc\" (UID: \"3fa637e0-ccb8-4713-a16b-f4c353d3cb8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc"
Nov 02 02:00:00.337518 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.337478    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqn5r\" (UniqueName: \"kubernetes.io/projected/5184916a-2d9b-40cc-9872-9fb0cd055b41-kube-api-access-xqn5r\") pod \"maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt\" (UID: \"5184916a-2d9b-40cc-9872-9fb0cd055b41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt"
Nov 02 02:00:00.337718 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.337678    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdv2m\" (UniqueName: \"kubernetes.io/projected/5dabb512-1926-413b-87ab-2a9c87e685f7-kube-api-access-kdv2m\") pod \"maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7\" (UID: \"5dabb512-1926-413b-87ab-2a9c87e685f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7"
Nov 02 02:00:00.337910 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.337881    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvfsp\" (UniqueName: \"kubernetes.io/projected/3fa637e0-ccb8-4713-a16b-f4c353d3cb8f-kube-api-access-xvfsp\") pod \"maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc\" (UID: \"3fa637e0-ccb8-4713-a16b-f4c353d3cb8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc"
Nov 02 02:00:00.338151 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.338080    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa637e0-ccb8-4713-a16b-f4c353d3cb8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc\" (UID: \"3fa637e0-ccb8-4713-a16b-f4c353d3cb8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc"
Nov 02 02:00:00.338151 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.338111    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g5sp\" (UniqueName: \"kubernetes.io/projected/2c0d592c-c444-4131-ac9c-13e3980e5a6d-kube-api-access-8g5sp\") pod \"maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2\" (UID: \"2c0d592c-c444-4131-ac9c-13e3980e5a6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2"
Nov 02 02:00:00.338401 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.338221    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtjsm\" (UniqueName: \"kubernetes.io/projected/3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e-kube-api-access-gtjsm\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm\" (UID: \"3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm"
Nov 02 02:00:00.338401 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.338262    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvb2w\" (UniqueName: \"kubernetes.io/projected/e5b031ba-6e9e-4852-abb4-218ab35f22b9-kube-api-access-tvb2w\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps\" (UID: \"e5b031ba-6e9e-4852-abb4-218ab35f22b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps"
Nov 02 02:00:00.338526 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.338431    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b031ba-6e9e-4852-abb4-218ab35f22b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps\" (UID: \"e5b031ba-6e9e-4852-abb4-218ab35f22b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps"
Nov 02 02:00:00.338526 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.338499    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dabb512-1926-413b-87ab-2a9c87e685f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7\" (UID: \"5dabb512-1926-413b-87ab-2a9c87e685f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7"
Nov 02 02:00:00.338618 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.338567    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b8bd\" (UniqueName: \"kubernetes.io/projected/d96d25d2-704b-40d5-9a6b-116c4eb3a4a4-kube-api-access-5b8bd\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh\" (UID: \"d96d25d2-704b-40d5-9a6b-116c4eb3a4a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh"
Nov 02 02:00:00.338618 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.338603    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0720184c-4f19-41d9-871f-b3a208201818-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9\" (UID: \"0720184c-4f19-41d9-871f-b3a208201818\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9"
Nov 02 02:00:00.338829 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.338641    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqrcl\" (UniqueName: \"kubernetes.io/projected/0720184c-4f19-41d9-871f-b3a208201818-kube-api-access-qqrcl\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9\" (UID: \"0720184c-4f19-41d9-871f-b3a208201818\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9"
Nov 02 02:00:00.338829 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.338672    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d96d25d2-704b-40d5-9a6b-116c4eb3a4a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh\" (UID: \"d96d25d2-704b-40d5-9a6b-116c4eb3a4a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh"
Nov 02 02:00:00.338829 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.338725    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c0d592c-c444-4131-ac9c-13e3980e5a6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2\" (UID: \"2c0d592c-c444-4131-ac9c-13e3980e5a6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2"
Nov 02 02:00:00.338829 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.338773    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/113179bd-c2bc-4fe6-909e-e1387c459bb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z\" (UID: \"113179bd-c2bc-4fe6-909e-e1387c459bb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z"
Nov 02 02:00:00.338829 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.338802    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqsqq\" (UniqueName: \"kubernetes.io/projected/3f5659fe-839b-4498-b9eb-2b76d6631417-kube-api-access-gqsqq\") pod \"maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw\" (UID: \"3f5659fe-839b-4498-b9eb-2b76d6631417\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw"
Nov 02 02:00:00.339102 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.338842    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqdcz\" (UniqueName: \"kubernetes.io/projected/b669073a-3244-4c17-9255-197abd78c544-kube-api-access-qqdcz\") pod \"maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498\" (UID: \"b669073a-3244-4c17-9255-197abd78c544\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498"
Nov 02 02:00:00.339102 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.338911    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdzsz\" (UniqueName: \"kubernetes.io/projected/2aefa8c8-6056-4fd0-8e70-739dbd421750-kube-api-access-zdzsz\") pod \"maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc\" (UID: \"2aefa8c8-6056-4fd0-8e70-739dbd421750\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc"
Nov 02 02:00:00.339102 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.338959    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glk7f\" (UniqueName: \"kubernetes.io/projected/113179bd-c2bc-4fe6-909e-e1387c459bb9-kube-api-access-glk7f\") pod \"maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z\" (UID: \"113179bd-c2bc-4fe6-909e-e1387c459bb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z"
Nov 02 02:00:00.339102 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.339022    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5184916a-2d9b-40cc-9872-9fb0cd055b41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt\" (UID: \"5184916a-2d9b-40cc-9872-9fb0cd055b41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt"
Nov 02 02:00:00.339102 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.339058    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f5659fe-839b-4498-b9eb-2b76d6631417-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw\" (UID: \"3f5659fe-839b-4498-b9eb-2b76d6631417\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw"
Nov 02 02:00:00.342532 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.339100    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aefa8c8-6056-4fd0-8e70-739dbd421750-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc\" (UID: \"2aefa8c8-6056-4fd0-8e70-739dbd421750\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc"
Nov 02 02:00:00.342532 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.339483    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dabb512-1926-413b-87ab-2a9c87e685f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7\" (UID: \"5dabb512-1926-413b-87ab-2a9c87e685f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7"
Nov 02 02:00:00.355436 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.355335    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9"]
Nov 02 02:00:00.372995 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.372621    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh"]
Nov 02 02:00:00.377952 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.377925    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc"]
Nov 02 02:00:00.399883 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.399855    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdv2m\" (UniqueName: \"kubernetes.io/projected/5dabb512-1926-413b-87ab-2a9c87e685f7-kube-api-access-kdv2m\") pod \"maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7\" (UID: \"5dabb512-1926-413b-87ab-2a9c87e685f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7"
Nov 02 02:00:00.402855 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.402827    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvfsp\" (UniqueName: \"kubernetes.io/projected/3fa637e0-ccb8-4713-a16b-f4c353d3cb8f-kube-api-access-xvfsp\") pod \"maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc\" (UID: \"3fa637e0-ccb8-4713-a16b-f4c353d3cb8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc"
Nov 02 02:00:00.440580 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.440554    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm\" (UID: \"3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm"
Nov 02 02:00:00.440714 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.440595    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b669073a-3244-4c17-9255-197abd78c544-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498\" (UID: \"b669073a-3244-4c17-9255-197abd78c544\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498"
Nov 02 02:00:00.440714 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.440652    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqn5r\" (UniqueName: \"kubernetes.io/projected/5184916a-2d9b-40cc-9872-9fb0cd055b41-kube-api-access-xqn5r\") pod \"maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt\" (UID: \"5184916a-2d9b-40cc-9872-9fb0cd055b41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt"
Nov 02 02:00:00.440714 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.440708    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g5sp\" (UniqueName: \"kubernetes.io/projected/2c0d592c-c444-4131-ac9c-13e3980e5a6d-kube-api-access-8g5sp\") pod \"maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2\" (UID: \"2c0d592c-c444-4131-ac9c-13e3980e5a6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2"
Nov 02 02:00:00.440886 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.440734    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtjsm\" (UniqueName: \"kubernetes.io/projected/3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e-kube-api-access-gtjsm\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm\" (UID: \"3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm"
Nov 02 02:00:00.440886 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.440754    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvb2w\" (UniqueName: \"kubernetes.io/projected/e5b031ba-6e9e-4852-abb4-218ab35f22b9-kube-api-access-tvb2w\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps\" (UID: \"e5b031ba-6e9e-4852-abb4-218ab35f22b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps"
Nov 02 02:00:00.440886 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.440787    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b031ba-6e9e-4852-abb4-218ab35f22b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps\" (UID: \"e5b031ba-6e9e-4852-abb4-218ab35f22b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps"
Nov 02 02:00:00.440886 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.440852    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b8bd\" (UniqueName: \"kubernetes.io/projected/d96d25d2-704b-40d5-9a6b-116c4eb3a4a4-kube-api-access-5b8bd\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh\" (UID: \"d96d25d2-704b-40d5-9a6b-116c4eb3a4a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh"
Nov 02 02:00:00.440886 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.440875    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0720184c-4f19-41d9-871f-b3a208201818-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9\" (UID: \"0720184c-4f19-41d9-871f-b3a208201818\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9"
Nov 02 02:00:00.441164 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.440906    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqrcl\" (UniqueName: \"kubernetes.io/projected/0720184c-4f19-41d9-871f-b3a208201818-kube-api-access-qqrcl\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9\" (UID: \"0720184c-4f19-41d9-871f-b3a208201818\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9"
Nov 02 02:00:00.441164 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.440937    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d96d25d2-704b-40d5-9a6b-116c4eb3a4a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh\" (UID: \"d96d25d2-704b-40d5-9a6b-116c4eb3a4a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh"
Nov 02 02:00:00.441164 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.440975    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c0d592c-c444-4131-ac9c-13e3980e5a6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2\" (UID: \"2c0d592c-c444-4131-ac9c-13e3980e5a6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2"
Nov 02 02:00:00.441164 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.441022    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/113179bd-c2bc-4fe6-909e-e1387c459bb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z\" (UID: \"113179bd-c2bc-4fe6-909e-e1387c459bb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z"
Nov 02 02:00:00.441164 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.441047    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqsqq\" (UniqueName: \"kubernetes.io/projected/3f5659fe-839b-4498-b9eb-2b76d6631417-kube-api-access-gqsqq\") pod \"maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw\" (UID: \"3f5659fe-839b-4498-b9eb-2b76d6631417\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw"
Nov 02 02:00:00.441164 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.441086    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqdcz\" (UniqueName: \"kubernetes.io/projected/b669073a-3244-4c17-9255-197abd78c544-kube-api-access-qqdcz\") pod \"maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498\" (UID: \"b669073a-3244-4c17-9255-197abd78c544\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498"
Nov 02 02:00:00.441499 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.441324    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdzsz\" (UniqueName: \"kubernetes.io/projected/2aefa8c8-6056-4fd0-8e70-739dbd421750-kube-api-access-zdzsz\") pod \"maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc\" (UID: \"2aefa8c8-6056-4fd0-8e70-739dbd421750\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc"
Nov 02 02:00:00.441499 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.441371    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glk7f\" (UniqueName: \"kubernetes.io/projected/113179bd-c2bc-4fe6-909e-e1387c459bb9-kube-api-access-glk7f\") pod \"maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z\" (UID: \"113179bd-c2bc-4fe6-909e-e1387c459bb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z"
Nov 02 02:00:00.441499 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.441407    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5184916a-2d9b-40cc-9872-9fb0cd055b41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt\" (UID: \"5184916a-2d9b-40cc-9872-9fb0cd055b41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt"
Nov 02 02:00:00.441499 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.441431    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f5659fe-839b-4498-b9eb-2b76d6631417-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw\" (UID: \"3f5659fe-839b-4498-b9eb-2b76d6631417\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw"
Nov 02 02:00:00.441499 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.441462    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aefa8c8-6056-4fd0-8e70-739dbd421750-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc\" (UID: \"2aefa8c8-6056-4fd0-8e70-739dbd421750\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc"
Nov 02 02:00:00.441754 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.441710    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0720184c-4f19-41d9-871f-b3a208201818-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9\" (UID: \"0720184c-4f19-41d9-871f-b3a208201818\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9"
Nov 02 02:00:00.441812 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.441770    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b031ba-6e9e-4852-abb4-218ab35f22b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps\" (UID: \"e5b031ba-6e9e-4852-abb4-218ab35f22b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps"
Nov 02 02:00:00.441812 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.441782    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d96d25d2-704b-40d5-9a6b-116c4eb3a4a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh\" (UID: \"d96d25d2-704b-40d5-9a6b-116c4eb3a4a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh"
Nov 02 02:00:00.441992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.441955    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c0d592c-c444-4131-ac9c-13e3980e5a6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2\" (UID: \"2c0d592c-c444-4131-ac9c-13e3980e5a6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2"
Nov 02 02:00:00.442137 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.442112    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/113179bd-c2bc-4fe6-909e-e1387c459bb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z\" (UID: \"113179bd-c2bc-4fe6-909e-e1387c459bb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z"
Nov 02 02:00:00.442229 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.442161    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aefa8c8-6056-4fd0-8e70-739dbd421750-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc\" (UID: \"2aefa8c8-6056-4fd0-8e70-739dbd421750\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc"
Nov 02 02:00:00.442377 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.442355    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b669073a-3244-4c17-9255-197abd78c544-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498\" (UID: \"b669073a-3244-4c17-9255-197abd78c544\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498"
Nov 02 02:00:00.442429 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.442366    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm\" (UID: \"3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm"
Nov 02 02:00:00.442476 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.442460    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5184916a-2d9b-40cc-9872-9fb0cd055b41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt\" (UID: \"5184916a-2d9b-40cc-9872-9fb0cd055b41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt"
Nov 02 02:00:00.442544 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.442524    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f5659fe-839b-4498-b9eb-2b76d6631417-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw\" (UID: \"3f5659fe-839b-4498-b9eb-2b76d6631417\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw"
Nov 02 02:00:00.480238 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.480210    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5"]
Nov 02 02:00:00.480782 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:00:00.480756    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f9e832d_76eb_41c1_91b9_db784e55c47e.slice/crio-50f29fc49d204725524827acecfee66f97af1d6d7bb19686f029ffa09451c109 WatchSource:0}: Error finding container 50f29fc49d204725524827acecfee66f97af1d6d7bb19686f029ffa09451c109: Status 404 returned error can't find the container with id 50f29fc49d204725524827acecfee66f97af1d6d7bb19686f029ffa09451c109
Nov 02 02:00:00.503059 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.503032    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtjsm\" (UniqueName: \"kubernetes.io/projected/3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e-kube-api-access-gtjsm\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm\" (UID: \"3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm"
Nov 02 02:00:00.503229 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.503204    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g5sp\" (UniqueName: \"kubernetes.io/projected/2c0d592c-c444-4131-ac9c-13e3980e5a6d-kube-api-access-8g5sp\") pod \"maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2\" (UID: \"2c0d592c-c444-4131-ac9c-13e3980e5a6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2"
Nov 02 02:00:00.503533 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.503511    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdzsz\" (UniqueName: \"kubernetes.io/projected/2aefa8c8-6056-4fd0-8e70-739dbd421750-kube-api-access-zdzsz\") pod \"maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc\" (UID: \"2aefa8c8-6056-4fd0-8e70-739dbd421750\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc"
Nov 02 02:00:00.503621 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.503575    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glk7f\" (UniqueName: \"kubernetes.io/projected/113179bd-c2bc-4fe6-909e-e1387c459bb9-kube-api-access-glk7f\") pod \"maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z\" (UID: \"113179bd-c2bc-4fe6-909e-e1387c459bb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z"
Nov 02 02:00:00.503689 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.503646    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqrcl\" (UniqueName: \"kubernetes.io/projected/0720184c-4f19-41d9-871f-b3a208201818-kube-api-access-qqrcl\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9\" (UID: \"0720184c-4f19-41d9-871f-b3a208201818\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9"
Nov 02 02:00:00.504559 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.504534    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvb2w\" (UniqueName: \"kubernetes.io/projected/e5b031ba-6e9e-4852-abb4-218ab35f22b9-kube-api-access-tvb2w\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps\" (UID: \"e5b031ba-6e9e-4852-abb4-218ab35f22b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps"
Nov 02 02:00:00.504658 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.504622    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b8bd\" (UniqueName: \"kubernetes.io/projected/d96d25d2-704b-40d5-9a6b-116c4eb3a4a4-kube-api-access-5b8bd\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh\" (UID: \"d96d25d2-704b-40d5-9a6b-116c4eb3a4a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh"
Nov 02 02:00:00.504976 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.504945    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqn5r\" (UniqueName: \"kubernetes.io/projected/5184916a-2d9b-40cc-9872-9fb0cd055b41-kube-api-access-xqn5r\") pod \"maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt\" (UID: \"5184916a-2d9b-40cc-9872-9fb0cd055b41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt"
Nov 02 02:00:00.505436 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.505413    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqsqq\" (UniqueName: \"kubernetes.io/projected/3f5659fe-839b-4498-b9eb-2b76d6631417-kube-api-access-gqsqq\") pod \"maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw\" (UID: \"3f5659fe-839b-4498-b9eb-2b76d6631417\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw"
Nov 02 02:00:00.506117 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.506098    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqdcz\" (UniqueName: \"kubernetes.io/projected/b669073a-3244-4c17-9255-197abd78c544-kube-api-access-qqdcz\") pod \"maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498\" (UID: \"b669073a-3244-4c17-9255-197abd78c544\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498"
Nov 02 02:00:00.563732 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.563709    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc"
Nov 02 02:00:00.594467 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.594406    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7"
Nov 02 02:00:00.601065 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.601041    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw"
Nov 02 02:00:00.657814 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.657560    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps"
Nov 02 02:00:00.665257 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.665089    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2"
Nov 02 02:00:00.685766 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.685566    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498"
Nov 02 02:00:00.689424 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.689359    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt"
Nov 02 02:00:00.698302 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.698101    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z"
Nov 02 02:00:00.708091 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.707900    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9"
Nov 02 02:00:00.717880 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.717700    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh"
Nov 02 02:00:00.726717 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.725500    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc"
Nov 02 02:00:00.748784 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.731832    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm"
Nov 02 02:00:00.857598 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.857504    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk" event={"ID":"05e9bda3-da6b-437a-9b42-3a9a9a4aafcd","Type":"ContainerStarted","Data":"f09bb8eb4648d9c8e830d25a7f36dbec1ed0702c35cc6a675c1d8fe564e41101"}
Nov 02 02:00:00.857835 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.857815    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk" event={"ID":"05e9bda3-da6b-437a-9b42-3a9a9a4aafcd","Type":"ContainerStarted","Data":"93e70366ba65f96d327e92b1de4c4a1c7922c9e5e9e891f78f0b7a24a317e5b9"}
Nov 02 02:00:00.858996 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.858974    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk"
Nov 02 02:00:00.861206 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.861184    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5" event={"ID":"1f9e832d-76eb-41c1-91b9-db784e55c47e","Type":"ContainerStarted","Data":"2bbfa81577528780b3d5cb20e0c63f6ef3a81d622cea2094b420903a1706d67f"}
Nov 02 02:00:00.861391 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.861373    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5" event={"ID":"1f9e832d-76eb-41c1-91b9-db784e55c47e","Type":"ContainerStarted","Data":"50f29fc49d204725524827acecfee66f97af1d6d7bb19686f029ffa09451c109"}
Nov 02 02:00:00.884099 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:00.865584    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5"
Nov 02 02:00:01.003187 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.002851    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc"]
Nov 02 02:00:01.005824 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:00:01.005706    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa637e0_ccb8_4713_a16b_f4c353d3cb8f.slice/crio-68cdefd866b12df0e366a7af869a29455b1462a6f93e9fcc75520baa817044ca WatchSource:0}: Error finding container 68cdefd866b12df0e366a7af869a29455b1462a6f93e9fcc75520baa817044ca: Status 404 returned error can't find the container with id 68cdefd866b12df0e366a7af869a29455b1462a6f93e9fcc75520baa817044ca
Nov 02 02:00:01.231670 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.231647    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw"]
Nov 02 02:00:01.307969 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:00:01.307940    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c0d592c_c444_4131_ac9c_13e3980e5a6d.slice/crio-a575ffb6c0157d2e5a0f1bcd78d696eba798bf726aa7b30ae005c58843a80037 WatchSource:0}: Error finding container a575ffb6c0157d2e5a0f1bcd78d696eba798bf726aa7b30ae005c58843a80037: Status 404 returned error can't find the container with id a575ffb6c0157d2e5a0f1bcd78d696eba798bf726aa7b30ae005c58843a80037
Nov 02 02:00:01.309198 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.309177    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2"]
Nov 02 02:00:01.319965 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.319941    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps"]
Nov 02 02:00:01.323254 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:00:01.323226    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b031ba_6e9e_4852_abb4_218ab35f22b9.slice/crio-7e7531b8cd3f0985e2328cf4078427861102161e8d7195bc74d969605e5a1bae WatchSource:0}: Error finding container 7e7531b8cd3f0985e2328cf4078427861102161e8d7195bc74d969605e5a1bae: Status 404 returned error can't find the container with id 7e7531b8cd3f0985e2328cf4078427861102161e8d7195bc74d969605e5a1bae
Nov 02 02:00:01.346961 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.346934    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498"]
Nov 02 02:00:01.351749 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:00:01.351720    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb669073a_3244_4c17_9255_197abd78c544.slice/crio-c2c3c7799e04d778a4c57e5aa48ca58770e8aea6800b1ec26e8b0968a5056e2c WatchSource:0}: Error finding container c2c3c7799e04d778a4c57e5aa48ca58770e8aea6800b1ec26e8b0968a5056e2c: Status 404 returned error can't find the container with id c2c3c7799e04d778a4c57e5aa48ca58770e8aea6800b1ec26e8b0968a5056e2c
Nov 02 02:00:01.404074 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.404048    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7"]
Nov 02 02:00:01.404744 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:00:01.404718    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dabb512_1926_413b_87ab_2a9c87e685f7.slice/crio-31226ed0840322d4ef0e44fa23c5df5c0b4750ba0a9b1bc8093a14857aae0d6c WatchSource:0}: Error finding container 31226ed0840322d4ef0e44fa23c5df5c0b4750ba0a9b1bc8093a14857aae0d6c: Status 404 returned error can't find the container with id 31226ed0840322d4ef0e44fa23c5df5c0b4750ba0a9b1bc8093a14857aae0d6c
Nov 02 02:00:01.431338 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.431308    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt"]
Nov 02 02:00:01.436710 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:00:01.436684    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5184916a_2d9b_40cc_9872_9fb0cd055b41.slice/crio-c501d6c2c1491cc54a611060df864366d1089b93cf67874f3c77dbf3a3c8099c WatchSource:0}: Error finding container c501d6c2c1491cc54a611060df864366d1089b93cf67874f3c77dbf3a3c8099c: Status 404 returned error can't find the container with id c501d6c2c1491cc54a611060df864366d1089b93cf67874f3c77dbf3a3c8099c
Nov 02 02:00:01.442525 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.442501    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z"]
Nov 02 02:00:01.442624 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:00:01.442536    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod113179bd_c2bc_4fe6_909e_e1387c459bb9.slice/crio-78e703413f17628e5048c0199784e641e7e2d3f69df70e5c21cdfb6bd76c94b0 WatchSource:0}: Error finding container 78e703413f17628e5048c0199784e641e7e2d3f69df70e5c21cdfb6bd76c94b0: Status 404 returned error can't find the container with id 78e703413f17628e5048c0199784e641e7e2d3f69df70e5c21cdfb6bd76c94b0
Nov 02 02:00:01.493978 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.493946    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc"]
Nov 02 02:00:01.514944 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.514917    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm"]
Nov 02 02:00:01.569217 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:00:01.569181    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aefa8c8_6056_4fd0_8e70_739dbd421750.slice/crio-4090dcba836307fb18c441d0feea248a40a4f3dfd2fd01de5184b409bc75c8f1 WatchSource:0}: Error finding container 4090dcba836307fb18c441d0feea248a40a4f3dfd2fd01de5184b409bc75c8f1: Status 404 returned error can't find the container with id 4090dcba836307fb18c441d0feea248a40a4f3dfd2fd01de5184b409bc75c8f1
Nov 02 02:00:01.587143 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.587108    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh"]
Nov 02 02:00:01.594145 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.594121    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9"]
Nov 02 02:00:01.594885 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:00:01.594847    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd96d25d2_704b_40d5_9a6b_116c4eb3a4a4.slice/crio-2d2ce0eb93a4940db1dc2aaf4bb980c207d073e41f2d441f911b2c2a36aa82c2 WatchSource:0}: Error finding container 2d2ce0eb93a4940db1dc2aaf4bb980c207d073e41f2d441f911b2c2a36aa82c2: Status 404 returned error can't find the container with id 2d2ce0eb93a4940db1dc2aaf4bb980c207d073e41f2d441f911b2c2a36aa82c2
Nov 02 02:00:01.597334 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:00:01.597269    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0720184c_4f19_41d9_871f_b3a208201818.slice/crio-a62d998b79621bf69270062c8627829799a8db7625564875aea5a7231e212548 WatchSource:0}: Error finding container a62d998b79621bf69270062c8627829799a8db7625564875aea5a7231e212548: Status 404 returned error can't find the container with id a62d998b79621bf69270062c8627829799a8db7625564875aea5a7231e212548
Nov 02 02:00:01.727666 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.727561    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5" podStartSLOduration=2.72754853 podStartE2EDuration="2.72754853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:01.72516858 +0000 UTC m=+2066.628850708" watchObservedRunningTime="2026-11-02 02:00:01.72754853 +0000 UTC m=+2066.631230643"
Nov 02 02:00:01.932475 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.927622    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk" podStartSLOduration=2.927599682 podStartE2EDuration="2.927599682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:01.855370096 +0000 UTC m=+2066.759052210" watchObservedRunningTime="2026-11-02 02:00:01.927599682 +0000 UTC m=+2066.831281795"
Nov 02 02:00:01.940771 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.938546    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw" event={"ID":"3f5659fe-839b-4498-b9eb-2b76d6631417","Type":"ContainerStarted","Data":"75e2ed2879b802fde38596ba031e620ae3a653b951c5af78ae1751d761cde0e0"}
Nov 02 02:00:01.941422 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.941393    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9" event={"ID":"0720184c-4f19-41d9-871f-b3a208201818","Type":"ContainerStarted","Data":"a62d998b79621bf69270062c8627829799a8db7625564875aea5a7231e212548"}
Nov 02 02:00:01.944541 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.944514    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc" event={"ID":"2aefa8c8-6056-4fd0-8e70-739dbd421750","Type":"ContainerStarted","Data":"4090dcba836307fb18c441d0feea248a40a4f3dfd2fd01de5184b409bc75c8f1"}
Nov 02 02:00:01.947658 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.947632    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498" event={"ID":"b669073a-3244-4c17-9255-197abd78c544","Type":"ContainerStarted","Data":"c2c3c7799e04d778a4c57e5aa48ca58770e8aea6800b1ec26e8b0968a5056e2c"}
Nov 02 02:00:01.950439 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.949079    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2" event={"ID":"2c0d592c-c444-4131-ac9c-13e3980e5a6d","Type":"ContainerStarted","Data":"a575ffb6c0157d2e5a0f1bcd78d696eba798bf726aa7b30ae005c58843a80037"}
Nov 02 02:00:01.950603 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.950579    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps" event={"ID":"e5b031ba-6e9e-4852-abb4-218ab35f22b9","Type":"ContainerStarted","Data":"7e7531b8cd3f0985e2328cf4078427861102161e8d7195bc74d969605e5a1bae"}
Nov 02 02:00:01.952077 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.951982    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7" event={"ID":"5dabb512-1926-413b-87ab-2a9c87e685f7","Type":"ContainerStarted","Data":"31226ed0840322d4ef0e44fa23c5df5c0b4750ba0a9b1bc8093a14857aae0d6c"}
Nov 02 02:00:01.953958 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.953933    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm" event={"ID":"3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e","Type":"ContainerStarted","Data":"11aee98cb5361b121615c74492d172988991fdf0e51c40dd84a83c122296a45f"}
Nov 02 02:00:01.955380 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.955341    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z" event={"ID":"113179bd-c2bc-4fe6-909e-e1387c459bb9","Type":"ContainerStarted","Data":"78e703413f17628e5048c0199784e641e7e2d3f69df70e5c21cdfb6bd76c94b0"}
Nov 02 02:00:01.957379 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.957358    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc" event={"ID":"3fa637e0-ccb8-4713-a16b-f4c353d3cb8f","Type":"ContainerStarted","Data":"d4a98ae5441c4dfb0480f9462d8b75d79c6ad9fd6fcaad246469b9020d22b70b"}
Nov 02 02:00:01.957465 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.957390    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc" event={"ID":"3fa637e0-ccb8-4713-a16b-f4c353d3cb8f","Type":"ContainerStarted","Data":"68cdefd866b12df0e366a7af869a29455b1462a6f93e9fcc75520baa817044ca"}
Nov 02 02:00:01.959067 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.959048    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh" event={"ID":"d96d25d2-704b-40d5-9a6b-116c4eb3a4a4","Type":"ContainerStarted","Data":"2d2ce0eb93a4940db1dc2aaf4bb980c207d073e41f2d441f911b2c2a36aa82c2"}
Nov 02 02:00:01.962497 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:01.962459    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt" event={"ID":"5184916a-2d9b-40cc-9872-9fb0cd055b41","Type":"ContainerStarted","Data":"c501d6c2c1491cc54a611060df864366d1089b93cf67874f3c77dbf3a3c8099c"}
Nov 02 02:00:02.092934 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.092884    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc" podStartSLOduration=2.092870045 podStartE2EDuration="2.092870045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:02.088519316 +0000 UTC m=+2066.992201432" watchObservedRunningTime="2026-11-02 02:00:02.092870045 +0000 UTC m=+2066.996552213"
Nov 02 02:00:02.649701 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:02.649665    5639 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dabb512_1926_413b_87ab_2a9c87e685f7.slice/crio-conmon-877bae349994e3ecf78baf44ebdaa18f9320dd7c34252d23d52fbe8d3b13371c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dabb512_1926_413b_87ab_2a9c87e685f7.slice/crio-877bae349994e3ecf78baf44ebdaa18f9320dd7c34252d23d52fbe8d3b13371c.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 02:00:02.975145 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.975109    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc" event={"ID":"2aefa8c8-6056-4fd0-8e70-739dbd421750","Type":"ContainerStarted","Data":"6c3e0a176a3eb641d85b65047f344486461b200b347b9b212c24e9cc2af2245c"}
Nov 02 02:00:02.977199 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.977172    5639 generic.go:356] "Generic (PLEG): container finished" podID="b669073a-3244-4c17-9255-197abd78c544" containerID="4a87c354091ecf33cc7b8c4e750e8f106c86e391f9c43d952a84d7eca2871605" exitCode=60
Nov 02 02:00:02.977313 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.977255    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498" event={"ID":"b669073a-3244-4c17-9255-197abd78c544","Type":"ContainerDied","Data":"4a87c354091ecf33cc7b8c4e750e8f106c86e391f9c43d952a84d7eca2871605"}
Nov 02 02:00:02.977538 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.977520    5639 scope.go:122] "RemoveContainer" containerID="4a87c354091ecf33cc7b8c4e750e8f106c86e391f9c43d952a84d7eca2871605"
Nov 02 02:00:02.979585 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.979549    5639 generic.go:356] "Generic (PLEG): container finished" podID="2c0d592c-c444-4131-ac9c-13e3980e5a6d" containerID="ad3a30088d63abd710e6181a992ae8b0d89808d3c76f6124fbeb42347a90f56a" exitCode=60
Nov 02 02:00:02.979691 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.979640    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2" event={"ID":"2c0d592c-c444-4131-ac9c-13e3980e5a6d","Type":"ContainerDied","Data":"ad3a30088d63abd710e6181a992ae8b0d89808d3c76f6124fbeb42347a90f56a"}
Nov 02 02:00:02.979885 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.979869    5639 scope.go:122] "RemoveContainer" containerID="ad3a30088d63abd710e6181a992ae8b0d89808d3c76f6124fbeb42347a90f56a"
Nov 02 02:00:02.981781 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.981640    5639 generic.go:356] "Generic (PLEG): container finished" podID="e5b031ba-6e9e-4852-abb4-218ab35f22b9" containerID="1c765e19ebf4c01bd1cc4991bb9f787f347ec6343c3f4a84b4f51822dc6fc606" exitCode=60
Nov 02 02:00:02.981781 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.981707    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps" event={"ID":"e5b031ba-6e9e-4852-abb4-218ab35f22b9","Type":"ContainerDied","Data":"1c765e19ebf4c01bd1cc4991bb9f787f347ec6343c3f4a84b4f51822dc6fc606"}
Nov 02 02:00:02.981991 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.981975    5639 scope.go:122] "RemoveContainer" containerID="1c765e19ebf4c01bd1cc4991bb9f787f347ec6343c3f4a84b4f51822dc6fc606"
Nov 02 02:00:02.983361 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.983337    5639 generic.go:356] "Generic (PLEG): container finished" podID="5dabb512-1926-413b-87ab-2a9c87e685f7" containerID="877bae349994e3ecf78baf44ebdaa18f9320dd7c34252d23d52fbe8d3b13371c" exitCode=60
Nov 02 02:00:02.983491 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.983474    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7" event={"ID":"5dabb512-1926-413b-87ab-2a9c87e685f7","Type":"ContainerDied","Data":"877bae349994e3ecf78baf44ebdaa18f9320dd7c34252d23d52fbe8d3b13371c"}
Nov 02 02:00:02.983731 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.983631    5639 scope.go:122] "RemoveContainer" containerID="877bae349994e3ecf78baf44ebdaa18f9320dd7c34252d23d52fbe8d3b13371c"
Nov 02 02:00:02.985095 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.985071    5639 generic.go:356] "Generic (PLEG): container finished" podID="3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e" containerID="f86d4f08df1fa1f8050b669099ce5c6e6227282172da8b49e6da5857ac40ae6f" exitCode=60
Nov 02 02:00:02.985179 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.985143    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm" event={"ID":"3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e","Type":"ContainerDied","Data":"f86d4f08df1fa1f8050b669099ce5c6e6227282172da8b49e6da5857ac40ae6f"}
Nov 02 02:00:02.985373 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.985357    5639 scope.go:122] "RemoveContainer" containerID="f86d4f08df1fa1f8050b669099ce5c6e6227282172da8b49e6da5857ac40ae6f"
Nov 02 02:00:02.986939 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.986919    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z" event={"ID":"113179bd-c2bc-4fe6-909e-e1387c459bb9","Type":"ContainerStarted","Data":"186bba4ef2bb145c8aaee34055f69e98e9fb9e87bbdd3ad3b14374d4b0ea45b9"}
Nov 02 02:00:02.992588 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.992506    5639 generic.go:356] "Generic (PLEG): container finished" podID="3fa637e0-ccb8-4713-a16b-f4c353d3cb8f" containerID="d4a98ae5441c4dfb0480f9462d8b75d79c6ad9fd6fcaad246469b9020d22b70b" exitCode=60
Nov 02 02:00:02.992588 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.992573    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc" event={"ID":"3fa637e0-ccb8-4713-a16b-f4c353d3cb8f","Type":"ContainerDied","Data":"d4a98ae5441c4dfb0480f9462d8b75d79c6ad9fd6fcaad246469b9020d22b70b"}
Nov 02 02:00:02.992881 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.992861    5639 scope.go:122] "RemoveContainer" containerID="d4a98ae5441c4dfb0480f9462d8b75d79c6ad9fd6fcaad246469b9020d22b70b"
Nov 02 02:00:02.995041 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.995019    5639 generic.go:356] "Generic (PLEG): container finished" podID="d96d25d2-704b-40d5-9a6b-116c4eb3a4a4" containerID="fb178a59d1d70a8828764ea256461c682265673fe0efebc29392bd56945ac675" exitCode=60
Nov 02 02:00:02.995142 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.995043    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh" event={"ID":"d96d25d2-704b-40d5-9a6b-116c4eb3a4a4","Type":"ContainerDied","Data":"fb178a59d1d70a8828764ea256461c682265673fe0efebc29392bd56945ac675"}
Nov 02 02:00:02.995610 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.995292    5639 scope.go:122] "RemoveContainer" containerID="fb178a59d1d70a8828764ea256461c682265673fe0efebc29392bd56945ac675"
Nov 02 02:00:02.996798 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.996779    5639 generic.go:356] "Generic (PLEG): container finished" podID="5184916a-2d9b-40cc-9872-9fb0cd055b41" containerID="c8ca4232a8550c7f0de63fcef4e3a8c1086f17479dd814d934d96ea7561c7d2a" exitCode=60
Nov 02 02:00:02.996881 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.996861    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt" event={"ID":"5184916a-2d9b-40cc-9872-9fb0cd055b41","Type":"ContainerDied","Data":"c8ca4232a8550c7f0de63fcef4e3a8c1086f17479dd814d934d96ea7561c7d2a"}
Nov 02 02:00:02.997118 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.997102    5639 scope.go:122] "RemoveContainer" containerID="c8ca4232a8550c7f0de63fcef4e3a8c1086f17479dd814d934d96ea7561c7d2a"
Nov 02 02:00:02.998692 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.998674    5639 generic.go:356] "Generic (PLEG): container finished" podID="3f5659fe-839b-4498-b9eb-2b76d6631417" containerID="f05341573613f7bc126704fd903bd1cb84b335bfa496651464d6e24e4913660f" exitCode=60
Nov 02 02:00:02.998943 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.998894    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw" event={"ID":"3f5659fe-839b-4498-b9eb-2b76d6631417","Type":"ContainerDied","Data":"f05341573613f7bc126704fd903bd1cb84b335bfa496651464d6e24e4913660f"}
Nov 02 02:00:02.999041 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:02.999024    5639 scope.go:122] "RemoveContainer" containerID="f05341573613f7bc126704fd903bd1cb84b335bfa496651464d6e24e4913660f"
Nov 02 02:00:03.010063 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:03.008361    5639 generic.go:356] "Generic (PLEG): container finished" podID="0720184c-4f19-41d9-871f-b3a208201818" containerID="dcad8c3e2a39835e444e9b051a3907a319781018bfb038d01fc92915ac247844" exitCode=60
Nov 02 02:00:03.010063 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:03.009107    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9" event={"ID":"0720184c-4f19-41d9-871f-b3a208201818","Type":"ContainerDied","Data":"dcad8c3e2a39835e444e9b051a3907a319781018bfb038d01fc92915ac247844"}
Nov 02 02:00:03.010063 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:03.009184    5639 scope.go:122] "RemoveContainer" containerID="dcad8c3e2a39835e444e9b051a3907a319781018bfb038d01fc92915ac247844"
Nov 02 02:00:03.096940 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:03.096895    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc" podStartSLOduration=3.096880356 podStartE2EDuration="3.096880356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:03.090127981 +0000 UTC m=+2067.993810120" watchObservedRunningTime="2026-11-02 02:00:03.096880356 +0000 UTC m=+2068.000562470"
Nov 02 02:00:03.585041 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:03.584995    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z" podStartSLOduration=3.5849827420000002 podStartE2EDuration="3.584982742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:03.581336328 +0000 UTC m=+2068.485018441" watchObservedRunningTime="2026-11-02 02:00:03.584982742 +0000 UTC m=+2068.488664854"
Nov 02 02:00:04.020008 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:04.019909    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc" event={"ID":"3fa637e0-ccb8-4713-a16b-f4c353d3cb8f","Type":"ContainerStarted","Data":"976ac0649baf50eb48a1d2c1011b46799f1ce7704a2fe9aecc96c64050f9d5c4"}
Nov 02 02:00:04.021981 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:04.021956    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh" event={"ID":"d96d25d2-704b-40d5-9a6b-116c4eb3a4a4","Type":"ContainerStarted","Data":"ab7123e22359bc51f700ef735c6816bf90e6653b8ef9b056061cdddf5aecfe1b"}
Nov 02 02:00:04.024057 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:04.024032    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt" event={"ID":"5184916a-2d9b-40cc-9872-9fb0cd055b41","Type":"ContainerStarted","Data":"9d95ede9faf2e2a5de00dad4e9a0120c11cf22a215e0fd5ee32f8e00e973851f"}
Nov 02 02:00:04.025970 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:04.025942    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw" event={"ID":"3f5659fe-839b-4498-b9eb-2b76d6631417","Type":"ContainerStarted","Data":"462817b959ded5b42e207de08949a580dac6b4e5bc5904469bfb66fa8a37eaa6"}
Nov 02 02:00:04.027968 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:04.027949    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9" event={"ID":"0720184c-4f19-41d9-871f-b3a208201818","Type":"ContainerStarted","Data":"27d514eb8a00e165dfce43f020cd0244297768ccbf1d8f80591501135b33bb5b"}
Nov 02 02:00:04.029517 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:04.029497    5639 generic.go:356] "Generic (PLEG): container finished" podID="2aefa8c8-6056-4fd0-8e70-739dbd421750" containerID="6c3e0a176a3eb641d85b65047f344486461b200b347b9b212c24e9cc2af2245c" exitCode=60
Nov 02 02:00:04.029626 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:04.029569    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc" event={"ID":"2aefa8c8-6056-4fd0-8e70-739dbd421750","Type":"ContainerDied","Data":"6c3e0a176a3eb641d85b65047f344486461b200b347b9b212c24e9cc2af2245c"}
Nov 02 02:00:04.029749 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:04.029735    5639 scope.go:122] "RemoveContainer" containerID="6c3e0a176a3eb641d85b65047f344486461b200b347b9b212c24e9cc2af2245c"
Nov 02 02:00:04.031620 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:04.031595    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498" event={"ID":"b669073a-3244-4c17-9255-197abd78c544","Type":"ContainerStarted","Data":"f85a792f91661b37a0e2dd2d097d52c0f6fa50841b9a2395fc020314e0ec01c2"}
Nov 02 02:00:04.033353 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:04.033297    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2" event={"ID":"2c0d592c-c444-4131-ac9c-13e3980e5a6d","Type":"ContainerStarted","Data":"cebf987c48fe0c564b4c50bef01a88d6fc5591ff76589b4ead019b4e74c85a6a"}
Nov 02 02:00:04.035259 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:04.035235    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps" event={"ID":"e5b031ba-6e9e-4852-abb4-218ab35f22b9","Type":"ContainerStarted","Data":"edcf164a962f0f56673dea4e446d47f40052f6e84f9177571b151e1669e96672"}
Nov 02 02:00:04.037222 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:04.037196    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7" event={"ID":"5dabb512-1926-413b-87ab-2a9c87e685f7","Type":"ContainerStarted","Data":"649f0490dc7a01499109949b05f9d51a79dea32c61bb077ae54ca3e929cd7eb6"}
Nov 02 02:00:04.040191 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:04.040172    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm" event={"ID":"3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e","Type":"ContainerStarted","Data":"968beddf34969d722a12c637c77c54715305304ce161a683713e191a2ebbc92b"}
Nov 02 02:00:04.042150 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:04.042129    5639 generic.go:356] "Generic (PLEG): container finished" podID="113179bd-c2bc-4fe6-909e-e1387c459bb9" containerID="186bba4ef2bb145c8aaee34055f69e98e9fb9e87bbdd3ad3b14374d4b0ea45b9" exitCode=60
Nov 02 02:00:04.042231 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:04.042161    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z" event={"ID":"113179bd-c2bc-4fe6-909e-e1387c459bb9","Type":"ContainerDied","Data":"186bba4ef2bb145c8aaee34055f69e98e9fb9e87bbdd3ad3b14374d4b0ea45b9"}
Nov 02 02:00:04.042483 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:04.042469    5639 scope.go:122] "RemoveContainer" containerID="186bba4ef2bb145c8aaee34055f69e98e9fb9e87bbdd3ad3b14374d4b0ea45b9"
Nov 02 02:00:04.507402 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:04.507351    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh" podStartSLOduration=4.5073373629999995 podStartE2EDuration="4.507337363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:04.504210558 +0000 UTC m=+2069.407892672" watchObservedRunningTime="2026-11-02 02:00:04.507337363 +0000 UTC m=+2069.411019499"
Nov 02 02:00:04.622248 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:04.622160    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498" podStartSLOduration=4.622145982 podStartE2EDuration="4.622145982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:04.620062517 +0000 UTC m=+2069.523744631" watchObservedRunningTime="2026-11-02 02:00:04.622145982 +0000 UTC m=+2069.525828093"
Nov 02 02:00:04.738529 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:04.738474    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps" podStartSLOduration=4.738460912 podStartE2EDuration="4.738460912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:04.734104299 +0000 UTC m=+2069.637786414" watchObservedRunningTime="2026-11-02 02:00:04.738460912 +0000 UTC m=+2069.642143073"
Nov 02 02:00:05.053819 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.053790    5639 generic.go:356] "Generic (PLEG): container finished" podID="3f5659fe-839b-4498-b9eb-2b76d6631417" containerID="462817b959ded5b42e207de08949a580dac6b4e5bc5904469bfb66fa8a37eaa6" exitCode=60
Nov 02 02:00:05.054242 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.053874    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw" event={"ID":"3f5659fe-839b-4498-b9eb-2b76d6631417","Type":"ContainerDied","Data":"462817b959ded5b42e207de08949a580dac6b4e5bc5904469bfb66fa8a37eaa6"}
Nov 02 02:00:05.054242 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.053941    5639 scope.go:122] "RemoveContainer" containerID="f05341573613f7bc126704fd903bd1cb84b335bfa496651464d6e24e4913660f"
Nov 02 02:00:05.054242 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.054214    5639 scope.go:122] "RemoveContainer" containerID="462817b959ded5b42e207de08949a580dac6b4e5bc5904469bfb66fa8a37eaa6"
Nov 02 02:00:05.054697 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:05.054671    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw_odh-ai-gateway-infra(3f5659fe-839b-4498-b9eb-2b76d6631417)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw" podUID="3f5659fe-839b-4498-b9eb-2b76d6631417"
Nov 02 02:00:05.056086 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.056064    5639 generic.go:356] "Generic (PLEG): container finished" podID="0720184c-4f19-41d9-871f-b3a208201818" containerID="27d514eb8a00e165dfce43f020cd0244297768ccbf1d8f80591501135b33bb5b" exitCode=60
Nov 02 02:00:05.056188 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.056134    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9" event={"ID":"0720184c-4f19-41d9-871f-b3a208201818","Type":"ContainerDied","Data":"27d514eb8a00e165dfce43f020cd0244297768ccbf1d8f80591501135b33bb5b"}
Nov 02 02:00:05.056434 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.056420    5639 scope.go:122] "RemoveContainer" containerID="27d514eb8a00e165dfce43f020cd0244297768ccbf1d8f80591501135b33bb5b"
Nov 02 02:00:05.056803 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:05.056766    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9_odh-ai-gateway-infra(0720184c-4f19-41d9-871f-b3a208201818)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9" podUID="0720184c-4f19-41d9-871f-b3a208201818"
Nov 02 02:00:05.058168 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.058141    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc" event={"ID":"2aefa8c8-6056-4fd0-8e70-739dbd421750","Type":"ContainerStarted","Data":"741979e020bee66ec59e9d4c438e8ea0462f013ae6183547a9aca528fda88980"}
Nov 02 02:00:05.060802 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.060526    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498" event={"ID":"b669073a-3244-4c17-9255-197abd78c544","Type":"ContainerDied","Data":"f85a792f91661b37a0e2dd2d097d52c0f6fa50841b9a2395fc020314e0ec01c2"}
Nov 02 02:00:05.060802 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.060631    5639 scope.go:122] "RemoveContainer" containerID="f85a792f91661b37a0e2dd2d097d52c0f6fa50841b9a2395fc020314e0ec01c2"
Nov 02 02:00:05.060978 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:05.060891    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498_odh-ai-gateway-infra(b669073a-3244-4c17-9255-197abd78c544)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498" podUID="b669073a-3244-4c17-9255-197abd78c544"
Nov 02 02:00:05.064349 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.060348    5639 generic.go:356] "Generic (PLEG): container finished" podID="b669073a-3244-4c17-9255-197abd78c544" containerID="f85a792f91661b37a0e2dd2d097d52c0f6fa50841b9a2395fc020314e0ec01c2" exitCode=60
Nov 02 02:00:05.066781 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.066754    5639 generic.go:356] "Generic (PLEG): container finished" podID="2c0d592c-c444-4131-ac9c-13e3980e5a6d" containerID="cebf987c48fe0c564b4c50bef01a88d6fc5591ff76589b4ead019b4e74c85a6a" exitCode=60
Nov 02 02:00:05.066986 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.066828    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2" event={"ID":"2c0d592c-c444-4131-ac9c-13e3980e5a6d","Type":"ContainerDied","Data":"cebf987c48fe0c564b4c50bef01a88d6fc5591ff76589b4ead019b4e74c85a6a"}
Nov 02 02:00:05.067052 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.067020    5639 scope.go:122] "RemoveContainer" containerID="cebf987c48fe0c564b4c50bef01a88d6fc5591ff76589b4ead019b4e74c85a6a"
Nov 02 02:00:05.067343 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:05.067257    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2_odh-ai-gateway-infra(2c0d592c-c444-4131-ac9c-13e3980e5a6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2" podUID="2c0d592c-c444-4131-ac9c-13e3980e5a6d"
Nov 02 02:00:05.069382 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.069361    5639 generic.go:356] "Generic (PLEG): container finished" podID="e5b031ba-6e9e-4852-abb4-218ab35f22b9" containerID="edcf164a962f0f56673dea4e446d47f40052f6e84f9177571b151e1669e96672" exitCode=60
Nov 02 02:00:05.069509 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.069434    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps" event={"ID":"e5b031ba-6e9e-4852-abb4-218ab35f22b9","Type":"ContainerDied","Data":"edcf164a962f0f56673dea4e446d47f40052f6e84f9177571b151e1669e96672"}
Nov 02 02:00:05.069666 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.069647    5639 scope.go:122] "RemoveContainer" containerID="edcf164a962f0f56673dea4e446d47f40052f6e84f9177571b151e1669e96672"
Nov 02 02:00:05.069904 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:05.069885    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps_odh-ai-gateway-infra(e5b031ba-6e9e-4852-abb4-218ab35f22b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps" podUID="e5b031ba-6e9e-4852-abb4-218ab35f22b9"
Nov 02 02:00:05.070985 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.070933    5639 scope.go:122] "RemoveContainer" containerID="dcad8c3e2a39835e444e9b051a3907a319781018bfb038d01fc92915ac247844"
Nov 02 02:00:05.072324 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.072271    5639 generic.go:356] "Generic (PLEG): container finished" podID="5dabb512-1926-413b-87ab-2a9c87e685f7" containerID="649f0490dc7a01499109949b05f9d51a79dea32c61bb077ae54ca3e929cd7eb6" exitCode=60
Nov 02 02:00:05.072407 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.072330    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7" event={"ID":"5dabb512-1926-413b-87ab-2a9c87e685f7","Type":"ContainerDied","Data":"649f0490dc7a01499109949b05f9d51a79dea32c61bb077ae54ca3e929cd7eb6"}
Nov 02 02:00:05.072597 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.072580    5639 scope.go:122] "RemoveContainer" containerID="649f0490dc7a01499109949b05f9d51a79dea32c61bb077ae54ca3e929cd7eb6"
Nov 02 02:00:05.072882 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:05.072859    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7_odh-ai-gateway-infra(5dabb512-1926-413b-87ab-2a9c87e685f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7" podUID="5dabb512-1926-413b-87ab-2a9c87e685f7"
Nov 02 02:00:05.074567 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.074531    5639 generic.go:356] "Generic (PLEG): container finished" podID="3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e" containerID="968beddf34969d722a12c637c77c54715305304ce161a683713e191a2ebbc92b" exitCode=60
Nov 02 02:00:05.074672 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.074581    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm" event={"ID":"3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e","Type":"ContainerDied","Data":"968beddf34969d722a12c637c77c54715305304ce161a683713e191a2ebbc92b"}
Nov 02 02:00:05.074816 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.074799    5639 scope.go:122] "RemoveContainer" containerID="968beddf34969d722a12c637c77c54715305304ce161a683713e191a2ebbc92b"
Nov 02 02:00:05.075068 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:05.075049    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm_odh-ai-gateway-infra(3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm" podUID="3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e"
Nov 02 02:00:05.077075 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.077034    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9" podStartSLOduration=5.077022811 podStartE2EDuration="5.077022811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:05.075690367 +0000 UTC m=+2069.979372480" watchObservedRunningTime="2026-11-02 02:00:05.077022811 +0000 UTC m=+2069.980704924"
Nov 02 02:00:05.081295 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.080398    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z" event={"ID":"113179bd-c2bc-4fe6-909e-e1387c459bb9","Type":"ContainerStarted","Data":"ed5d18c8969920ea383e38aff0a0dd2f9dcc1854ede9ee64e9085c3dd9a75c92"}
Nov 02 02:00:05.086974 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.086938    5639 generic.go:356] "Generic (PLEG): container finished" podID="3fa637e0-ccb8-4713-a16b-f4c353d3cb8f" containerID="976ac0649baf50eb48a1d2c1011b46799f1ce7704a2fe9aecc96c64050f9d5c4" exitCode=60
Nov 02 02:00:05.087077 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.086993    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc" event={"ID":"3fa637e0-ccb8-4713-a16b-f4c353d3cb8f","Type":"ContainerDied","Data":"976ac0649baf50eb48a1d2c1011b46799f1ce7704a2fe9aecc96c64050f9d5c4"}
Nov 02 02:00:05.087338 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.087273    5639 scope.go:122] "RemoveContainer" containerID="976ac0649baf50eb48a1d2c1011b46799f1ce7704a2fe9aecc96c64050f9d5c4"
Nov 02 02:00:05.087656 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:05.087631    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc_odh-ai-gateway-infra(3fa637e0-ccb8-4713-a16b-f4c353d3cb8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc" podUID="3fa637e0-ccb8-4713-a16b-f4c353d3cb8f"
Nov 02 02:00:05.090366 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.090345    5639 generic.go:356] "Generic (PLEG): container finished" podID="d96d25d2-704b-40d5-9a6b-116c4eb3a4a4" containerID="ab7123e22359bc51f700ef735c6816bf90e6653b8ef9b056061cdddf5aecfe1b" exitCode=60
Nov 02 02:00:05.090474 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.090426    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh" event={"ID":"d96d25d2-704b-40d5-9a6b-116c4eb3a4a4","Type":"ContainerDied","Data":"ab7123e22359bc51f700ef735c6816bf90e6653b8ef9b056061cdddf5aecfe1b"}
Nov 02 02:00:05.090656 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.090633    5639 scope.go:122] "RemoveContainer" containerID="ab7123e22359bc51f700ef735c6816bf90e6653b8ef9b056061cdddf5aecfe1b"
Nov 02 02:00:05.090897 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:05.090867    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh_odh-ai-gateway-infra(d96d25d2-704b-40d5-9a6b-116c4eb3a4a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh" podUID="d96d25d2-704b-40d5-9a6b-116c4eb3a4a4"
Nov 02 02:00:05.091672 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.091659    5639 scope.go:122] "RemoveContainer" containerID="4a87c354091ecf33cc7b8c4e750e8f106c86e391f9c43d952a84d7eca2871605"
Nov 02 02:00:05.092568 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.092539    5639 generic.go:356] "Generic (PLEG): container finished" podID="5184916a-2d9b-40cc-9872-9fb0cd055b41" containerID="9d95ede9faf2e2a5de00dad4e9a0120c11cf22a215e0fd5ee32f8e00e973851f" exitCode=60
Nov 02 02:00:05.092669 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.092596    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt" event={"ID":"5184916a-2d9b-40cc-9872-9fb0cd055b41","Type":"ContainerDied","Data":"9d95ede9faf2e2a5de00dad4e9a0120c11cf22a215e0fd5ee32f8e00e973851f"}
Nov 02 02:00:05.092877 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.092860    5639 scope.go:122] "RemoveContainer" containerID="9d95ede9faf2e2a5de00dad4e9a0120c11cf22a215e0fd5ee32f8e00e973851f"
Nov 02 02:00:05.093106 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:05.093084    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt_odh-ai-gateway-infra(5184916a-2d9b-40cc-9872-9fb0cd055b41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt" podUID="5184916a-2d9b-40cc-9872-9fb0cd055b41"
Nov 02 02:00:05.106006 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.105985    5639 scope.go:122] "RemoveContainer" containerID="ad3a30088d63abd710e6181a992ae8b0d89808d3c76f6124fbeb42347a90f56a"
Nov 02 02:00:05.120121 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.120102    5639 scope.go:122] "RemoveContainer" containerID="1c765e19ebf4c01bd1cc4991bb9f787f347ec6343c3f4a84b4f51822dc6fc606"
Nov 02 02:00:05.133764 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.133747    5639 scope.go:122] "RemoveContainer" containerID="877bae349994e3ecf78baf44ebdaa18f9320dd7c34252d23d52fbe8d3b13371c"
Nov 02 02:00:05.157358 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.157208    5639 scope.go:122] "RemoveContainer" containerID="f86d4f08df1fa1f8050b669099ce5c6e6227282172da8b49e6da5857ac40ae6f"
Nov 02 02:00:05.192471 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.192423    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2" podStartSLOduration=5.19240919 podStartE2EDuration="5.19240919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:05.190482528 +0000 UTC m=+2070.094164641" watchObservedRunningTime="2026-11-02 02:00:05.19240919 +0000 UTC m=+2070.096091303"
Nov 02 02:00:05.246969 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.246948    5639 scope.go:122] "RemoveContainer" containerID="d4a98ae5441c4dfb0480f9462d8b75d79c6ad9fd6fcaad246469b9020d22b70b"
Nov 02 02:00:05.261946 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.261925    5639 scope.go:122] "RemoveContainer" containerID="fb178a59d1d70a8828764ea256461c682265673fe0efebc29392bd56945ac675"
Nov 02 02:00:05.276262 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.276238    5639 scope.go:122] "RemoveContainer" containerID="c8ca4232a8550c7f0de63fcef4e3a8c1086f17479dd814d934d96ea7561c7d2a"
Nov 02 02:00:05.306256 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.306208    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7" podStartSLOduration=5.306195988 podStartE2EDuration="5.306195988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:05.305201475 +0000 UTC m=+2070.208883588" watchObservedRunningTime="2026-11-02 02:00:05.306195988 +0000 UTC m=+2070.209878100"
Nov 02 02:00:05.422604 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.422496    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm" podStartSLOduration=5.422482888 podStartE2EDuration="5.422482888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:05.419503945 +0000 UTC m=+2070.323186057" watchObservedRunningTime="2026-11-02 02:00:05.422482888 +0000 UTC m=+2070.326165002"
Nov 02 02:00:05.537531 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.537478    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt" podStartSLOduration=5.537463688 podStartE2EDuration="5.537463688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:05.533832783 +0000 UTC m=+2070.437514889" watchObservedRunningTime="2026-11-02 02:00:05.537463688 +0000 UTC m=+2070.441145801"
Nov 02 02:00:05.550395 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.550361    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r"]
Nov 02 02:00:05.651457 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.651412    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf"]
Nov 02 02:00:05.765497 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:05.764479    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw" podStartSLOduration=5.76446157 podStartE2EDuration="5.76446157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:05.764238723 +0000 UTC m=+2070.667920879" watchObservedRunningTime="2026-11-02 02:00:05.76446157 +0000 UTC m=+2070.668143683"
Nov 02 02:00:06.106625 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:06.106596    5639 scope.go:122] "RemoveContainer" containerID="968beddf34969d722a12c637c77c54715305304ce161a683713e191a2ebbc92b"
Nov 02 02:00:06.107035 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:06.106911    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm_odh-ai-gateway-infra(3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm" podUID="3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e"
Nov 02 02:00:06.108341 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:06.108320    5639 generic.go:356] "Generic (PLEG): container finished" podID="113179bd-c2bc-4fe6-909e-e1387c459bb9" containerID="ed5d18c8969920ea383e38aff0a0dd2f9dcc1854ede9ee64e9085c3dd9a75c92" exitCode=60
Nov 02 02:00:06.108456 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:06.108405    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z" event={"ID":"113179bd-c2bc-4fe6-909e-e1387c459bb9","Type":"ContainerDied","Data":"ed5d18c8969920ea383e38aff0a0dd2f9dcc1854ede9ee64e9085c3dd9a75c92"}
Nov 02 02:00:06.108507 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:06.108493    5639 scope.go:122] "RemoveContainer" containerID="186bba4ef2bb145c8aaee34055f69e98e9fb9e87bbdd3ad3b14374d4b0ea45b9"
Nov 02 02:00:06.108625 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:06.108605    5639 scope.go:122] "RemoveContainer" containerID="ed5d18c8969920ea383e38aff0a0dd2f9dcc1854ede9ee64e9085c3dd9a75c92"
Nov 02 02:00:06.108919 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:06.108890    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z_odh-ai-gateway-infra(113179bd-c2bc-4fe6-909e-e1387c459bb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z" podUID="113179bd-c2bc-4fe6-909e-e1387c459bb9"
Nov 02 02:00:06.110391 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:06.110368    5639 scope.go:122] "RemoveContainer" containerID="976ac0649baf50eb48a1d2c1011b46799f1ce7704a2fe9aecc96c64050f9d5c4"
Nov 02 02:00:06.110669 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:06.110648    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc_odh-ai-gateway-infra(3fa637e0-ccb8-4713-a16b-f4c353d3cb8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc" podUID="3fa637e0-ccb8-4713-a16b-f4c353d3cb8f"
Nov 02 02:00:06.111966 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:06.111939    5639 scope.go:122] "RemoveContainer" containerID="ab7123e22359bc51f700ef735c6816bf90e6653b8ef9b056061cdddf5aecfe1b"
Nov 02 02:00:06.112225 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:06.112207    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh_odh-ai-gateway-infra(d96d25d2-704b-40d5-9a6b-116c4eb3a4a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh" podUID="d96d25d2-704b-40d5-9a6b-116c4eb3a4a4"
Nov 02 02:00:06.113498 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:06.113480    5639 scope.go:122] "RemoveContainer" containerID="9d95ede9faf2e2a5de00dad4e9a0120c11cf22a215e0fd5ee32f8e00e973851f"
Nov 02 02:00:06.113721 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:06.113702    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt_odh-ai-gateway-infra(5184916a-2d9b-40cc-9872-9fb0cd055b41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt" podUID="5184916a-2d9b-40cc-9872-9fb0cd055b41"
Nov 02 02:00:06.116033 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:06.116015    5639 scope.go:122] "RemoveContainer" containerID="462817b959ded5b42e207de08949a580dac6b4e5bc5904469bfb66fa8a37eaa6"
Nov 02 02:00:06.116250 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:06.116225    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw_odh-ai-gateway-infra(3f5659fe-839b-4498-b9eb-2b76d6631417)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw" podUID="3f5659fe-839b-4498-b9eb-2b76d6631417"
Nov 02 02:00:06.117725 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:06.117709    5639 scope.go:122] "RemoveContainer" containerID="27d514eb8a00e165dfce43f020cd0244297768ccbf1d8f80591501135b33bb5b"
Nov 02 02:00:06.117944 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:06.117926    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9_odh-ai-gateway-infra(0720184c-4f19-41d9-871f-b3a208201818)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9" podUID="0720184c-4f19-41d9-871f-b3a208201818"
Nov 02 02:00:06.119780 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:06.119756    5639 generic.go:356] "Generic (PLEG): container finished" podID="2aefa8c8-6056-4fd0-8e70-739dbd421750" containerID="741979e020bee66ec59e9d4c438e8ea0462f013ae6183547a9aca528fda88980" exitCode=60
Nov 02 02:00:06.119865 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:06.119849    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc" event={"ID":"2aefa8c8-6056-4fd0-8e70-739dbd421750","Type":"ContainerDied","Data":"741979e020bee66ec59e9d4c438e8ea0462f013ae6183547a9aca528fda88980"}
Nov 02 02:00:06.120166 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:06.120150    5639 scope.go:122] "RemoveContainer" containerID="741979e020bee66ec59e9d4c438e8ea0462f013ae6183547a9aca528fda88980"
Nov 02 02:00:06.120434 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:06.120414    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc_odh-ai-gateway-infra(2aefa8c8-6056-4fd0-8e70-739dbd421750)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc" podUID="2aefa8c8-6056-4fd0-8e70-739dbd421750"
Nov 02 02:00:06.122184 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:06.122164    5639 scope.go:122] "RemoveContainer" containerID="f85a792f91661b37a0e2dd2d097d52c0f6fa50841b9a2395fc020314e0ec01c2"
Nov 02 02:00:06.122515 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:06.122493    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498_odh-ai-gateway-infra(b669073a-3244-4c17-9255-197abd78c544)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498" podUID="b669073a-3244-4c17-9255-197abd78c544"
Nov 02 02:00:06.124020 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:06.124004    5639 scope.go:122] "RemoveContainer" containerID="cebf987c48fe0c564b4c50bef01a88d6fc5591ff76589b4ead019b4e74c85a6a"
Nov 02 02:00:06.124253 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:06.124233    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2_odh-ai-gateway-infra(2c0d592c-c444-4131-ac9c-13e3980e5a6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2" podUID="2c0d592c-c444-4131-ac9c-13e3980e5a6d"
Nov 02 02:00:06.125527 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:06.125510    5639 scope.go:122] "RemoveContainer" containerID="6c3e0a176a3eb641d85b65047f344486461b200b347b9b212c24e9cc2af2245c"
Nov 02 02:00:06.125599 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:06.125582    5639 scope.go:122] "RemoveContainer" containerID="edcf164a962f0f56673dea4e446d47f40052f6e84f9177571b151e1669e96672"
Nov 02 02:00:06.125830 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:06.125809    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps_odh-ai-gateway-infra(e5b031ba-6e9e-4852-abb4-218ab35f22b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps" podUID="e5b031ba-6e9e-4852-abb4-218ab35f22b9"
Nov 02 02:00:06.127134 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:06.127116    5639 scope.go:122] "RemoveContainer" containerID="649f0490dc7a01499109949b05f9d51a79dea32c61bb077ae54ca3e929cd7eb6"
Nov 02 02:00:06.127364 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:06.127345    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7_odh-ai-gateway-infra(5dabb512-1926-413b-87ab-2a9c87e685f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7" podUID="5dabb512-1926-413b-87ab-2a9c87e685f7"
Nov 02 02:00:07.263872 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:07.255876    5639 scope.go:122] "RemoveContainer" containerID="741979e020bee66ec59e9d4c438e8ea0462f013ae6183547a9aca528fda88980"
Nov 02 02:00:07.263872 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:07.256315    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc_odh-ai-gateway-infra(2aefa8c8-6056-4fd0-8e70-739dbd421750)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc" podUID="2aefa8c8-6056-4fd0-8e70-739dbd421750"
Nov 02 02:00:07.263872 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:07.256502    5639 scope.go:122] "RemoveContainer" containerID="ed5d18c8969920ea383e38aff0a0dd2f9dcc1854ede9ee64e9085c3dd9a75c92"
Nov 02 02:00:07.263872 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:07.256806    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z_odh-ai-gateway-infra(113179bd-c2bc-4fe6-909e-e1387c459bb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z" podUID="113179bd-c2bc-4fe6-909e-e1387c459bb9"
Nov 02 02:00:09.096811 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:09.096775    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-109b07-q2hk7"]
Nov 02 02:00:09.103090 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:09.103073    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-109b07-q2hk7"
Nov 02 02:00:09.156086 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:09.156051    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:00:09.162571 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:09.162547    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-109b07-q2hk7"]
Nov 02 02:00:09.287214 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:09.287176    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba22028e-a74a-4b4c-baef-dd9502007685-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-109b07-q2hk7\" (UID: \"ba22028e-a74a-4b4c-baef-dd9502007685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-109b07-q2hk7"
Nov 02 02:00:09.287808 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:09.287779    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gcpp\" (UniqueName: \"kubernetes.io/projected/ba22028e-a74a-4b4c-baef-dd9502007685-kube-api-access-6gcpp\") pod \"maas-api-revoke-keys-e2e-api-a-109b07-q2hk7\" (UID: \"ba22028e-a74a-4b4c-baef-dd9502007685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-109b07-q2hk7"
Nov 02 02:00:09.287949 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:09.287854    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba22028e-a74a-4b4c-baef-dd9502007685-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-109b07-q2hk7\" (UID: \"ba22028e-a74a-4b4c-baef-dd9502007685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-109b07-q2hk7"
Nov 02 02:00:09.389073 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:09.388985    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gcpp\" (UniqueName: \"kubernetes.io/projected/ba22028e-a74a-4b4c-baef-dd9502007685-kube-api-access-6gcpp\") pod \"maas-api-revoke-keys-e2e-api-a-109b07-q2hk7\" (UID: \"ba22028e-a74a-4b4c-baef-dd9502007685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-109b07-q2hk7"
Nov 02 02:00:09.389073 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:09.389035    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba22028e-a74a-4b4c-baef-dd9502007685-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-109b07-q2hk7\" (UID: \"ba22028e-a74a-4b4c-baef-dd9502007685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-109b07-q2hk7"
Nov 02 02:00:09.389248 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:09.389137    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba22028e-a74a-4b4c-baef-dd9502007685-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-109b07-q2hk7\" (UID: \"ba22028e-a74a-4b4c-baef-dd9502007685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-109b07-q2hk7"
Nov 02 02:00:09.389769 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:09.389745    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba22028e-a74a-4b4c-baef-dd9502007685-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-109b07-q2hk7\" (UID: \"ba22028e-a74a-4b4c-baef-dd9502007685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-109b07-q2hk7"
Nov 02 02:00:09.391606 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:09.391587    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba22028e-a74a-4b4c-baef-dd9502007685-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-109b07-q2hk7\" (UID: \"ba22028e-a74a-4b4c-baef-dd9502007685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-109b07-q2hk7"
Nov 02 02:00:09.450475 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:09.450441    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gcpp\" (UniqueName: \"kubernetes.io/projected/ba22028e-a74a-4b4c-baef-dd9502007685-kube-api-access-6gcpp\") pod \"maas-api-revoke-keys-e2e-api-a-109b07-q2hk7\" (UID: \"ba22028e-a74a-4b4c-baef-dd9502007685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-109b07-q2hk7"
Nov 02 02:00:09.717516 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:09.717439    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-109b07-q2hk7"
Nov 02 02:00:10.102988 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:10.102961    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-109b07-q2hk7"]
Nov 02 02:00:10.103947 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:00:10.103924    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba22028e_a74a_4b4c_baef_dd9502007685.slice/crio-a17a8c4ff0ef797b70e2f5cccdd1196123ab1c423a528d710ef5a88ad382e32a WatchSource:0}: Error finding container a17a8c4ff0ef797b70e2f5cccdd1196123ab1c423a528d710ef5a88ad382e32a: Status 404 returned error can't find the container with id a17a8c4ff0ef797b70e2f5cccdd1196123ab1c423a528d710ef5a88ad382e32a
Nov 02 02:00:10.279997 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:10.279967    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-109b07-q2hk7" event={"ID":"ba22028e-a74a-4b4c-baef-dd9502007685","Type":"ContainerStarted","Data":"a17a8c4ff0ef797b70e2f5cccdd1196123ab1c423a528d710ef5a88ad382e32a"}
Nov 02 02:00:10.551218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:10.551180    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r" podUID="98bf5ee6-a486-456d-b9d8-d79bbaf5717e" containerName="payload-processing" containerID="cri-o://919752363bc6a1ce089d0858dbc15cf073b144a55b23d3228ce1a781e74141e8" gracePeriod=25
Nov 02 02:00:10.651935 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:10.651899    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf" podUID="1c1d68fb-24ac-454b-bf9c-02a894c43862" containerName="payload-pre-processing" containerID="cri-o://767d814c712ed1734f9e15a5792b601ec03c702557cdb508184d6ce3fc255e2b" gracePeriod=25
Nov 02 02:00:10.848233 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:10.848206    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r"
Nov 02 02:00:10.913166 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:10.913119    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98bf5ee6-a486-456d-b9d8-d79bbaf5717e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98bf5ee6-a486-456d-b9d8-d79bbaf5717e-plugins-config-volume\") pod \"98bf5ee6-a486-456d-b9d8-d79bbaf5717e\" (UID: \"98bf5ee6-a486-456d-b9d8-d79bbaf5717e\") "
Nov 02 02:00:10.913370 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:10.913238    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98bf5ee6-a486-456d-b9d8-d79bbaf5717e-kube-api-access-v2qsr\" (UniqueName: \"kubernetes.io/projected/98bf5ee6-a486-456d-b9d8-d79bbaf5717e-kube-api-access-v2qsr\") pod \"98bf5ee6-a486-456d-b9d8-d79bbaf5717e\" (UID: \"98bf5ee6-a486-456d-b9d8-d79bbaf5717e\") "
Nov 02 02:00:10.914066 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:10.913993    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98bf5ee6-a486-456d-b9d8-d79bbaf5717e-plugins-config-volume" pod "98bf5ee6-a486-456d-b9d8-d79bbaf5717e" (UID: "98bf5ee6-a486-456d-b9d8-d79bbaf5717e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:10.915855 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:10.915820    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98bf5ee6-a486-456d-b9d8-d79bbaf5717e-kube-api-access-v2qsr" pod "98bf5ee6-a486-456d-b9d8-d79bbaf5717e" (UID: "98bf5ee6-a486-456d-b9d8-d79bbaf5717e"). InnerVolumeSpecName "kube-api-access-v2qsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:10.937838 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:10.937812    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf"
Nov 02 02:00:11.020804 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.020710    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c1d68fb-24ac-454b-bf9c-02a894c43862-kube-api-access-9l92r\" (UniqueName: \"kubernetes.io/projected/1c1d68fb-24ac-454b-bf9c-02a894c43862-kube-api-access-9l92r\") pod \"1c1d68fb-24ac-454b-bf9c-02a894c43862\" (UID: \"1c1d68fb-24ac-454b-bf9c-02a894c43862\") "
Nov 02 02:00:11.020804 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.020783    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c1d68fb-24ac-454b-bf9c-02a894c43862-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c1d68fb-24ac-454b-bf9c-02a894c43862-plugins-config-volume\") pod \"1c1d68fb-24ac-454b-bf9c-02a894c43862\" (UID: \"1c1d68fb-24ac-454b-bf9c-02a894c43862\") "
Nov 02 02:00:11.021585 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.021546    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c1d68fb-24ac-454b-bf9c-02a894c43862-plugins-config-volume" pod "1c1d68fb-24ac-454b-bf9c-02a894c43862" (UID: "1c1d68fb-24ac-454b-bf9c-02a894c43862"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:11.021735 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.021640    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98bf5ee6-a486-456d-b9d8-d79bbaf5717e-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:11.021735 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.021663    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2qsr\" (UniqueName: \"kubernetes.io/projected/98bf5ee6-a486-456d-b9d8-d79bbaf5717e-kube-api-access-v2qsr\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:11.027710 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.026810    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c1d68fb-24ac-454b-bf9c-02a894c43862-kube-api-access-9l92r" pod "1c1d68fb-24ac-454b-bf9c-02a894c43862" (UID: "1c1d68fb-24ac-454b-bf9c-02a894c43862"). InnerVolumeSpecName "kube-api-access-9l92r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:11.124171 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.124146    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l92r\" (UniqueName: \"kubernetes.io/projected/1c1d68fb-24ac-454b-bf9c-02a894c43862-kube-api-access-9l92r\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:11.124171 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.124173    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c1d68fb-24ac-454b-bf9c-02a894c43862-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:11.289925 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.289851    5639 generic.go:356] "Generic (PLEG): container finished" podID="1c1d68fb-24ac-454b-bf9c-02a894c43862" containerID="767d814c712ed1734f9e15a5792b601ec03c702557cdb508184d6ce3fc255e2b" exitCode=0
Nov 02 02:00:11.289925 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.289899    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf"
Nov 02 02:00:11.290124 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.289938    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf" event={"ID":"1c1d68fb-24ac-454b-bf9c-02a894c43862","Type":"ContainerDied","Data":"767d814c712ed1734f9e15a5792b601ec03c702557cdb508184d6ce3fc255e2b"}
Nov 02 02:00:11.290124 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.289990    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf" event={"ID":"1c1d68fb-24ac-454b-bf9c-02a894c43862","Type":"ContainerDied","Data":"3f7faff5c2043534aec9afcdcc3d0a999cb6d8718bff239fa8b6e195b7e8be97"}
Nov 02 02:00:11.290124 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.290017    5639 scope.go:122] "RemoveContainer" containerID="767d814c712ed1734f9e15a5792b601ec03c702557cdb508184d6ce3fc255e2b"
Nov 02 02:00:11.292224 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.292196    5639 generic.go:356] "Generic (PLEG): container finished" podID="ba22028e-a74a-4b4c-baef-dd9502007685" containerID="937743b19555396824c40b60c446a181816440ec7214fef467d1e901274e2da9" exitCode=0
Nov 02 02:00:11.292611 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.292585    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-109b07-q2hk7" event={"ID":"ba22028e-a74a-4b4c-baef-dd9502007685","Type":"ContainerDied","Data":"937743b19555396824c40b60c446a181816440ec7214fef467d1e901274e2da9"}
Nov 02 02:00:11.294218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.294197    5639 generic.go:356] "Generic (PLEG): container finished" podID="98bf5ee6-a486-456d-b9d8-d79bbaf5717e" containerID="919752363bc6a1ce089d0858dbc15cf073b144a55b23d3228ce1a781e74141e8" exitCode=0
Nov 02 02:00:11.294361 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.294243    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r"
Nov 02 02:00:11.294361 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.294294    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r" event={"ID":"98bf5ee6-a486-456d-b9d8-d79bbaf5717e","Type":"ContainerDied","Data":"919752363bc6a1ce089d0858dbc15cf073b144a55b23d3228ce1a781e74141e8"}
Nov 02 02:00:11.294361 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.294349    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r" event={"ID":"98bf5ee6-a486-456d-b9d8-d79bbaf5717e","Type":"ContainerDied","Data":"9de058b74e8e9649c392c480730adf6af47cfba3629763721b789698ab5898b0"}
Nov 02 02:00:11.302790 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.302773    5639 scope.go:122] "RemoveContainer" containerID="767d814c712ed1734f9e15a5792b601ec03c702557cdb508184d6ce3fc255e2b"
Nov 02 02:00:11.303064 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:11.303045    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"767d814c712ed1734f9e15a5792b601ec03c702557cdb508184d6ce3fc255e2b\": container with ID starting with 767d814c712ed1734f9e15a5792b601ec03c702557cdb508184d6ce3fc255e2b not found: ID does not exist" containerID="767d814c712ed1734f9e15a5792b601ec03c702557cdb508184d6ce3fc255e2b"
Nov 02 02:00:11.303110 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.303073    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"767d814c712ed1734f9e15a5792b601ec03c702557cdb508184d6ce3fc255e2b"} err="failed to get container status \"767d814c712ed1734f9e15a5792b601ec03c702557cdb508184d6ce3fc255e2b\": rpc error: code = NotFound desc = could not find container \"767d814c712ed1734f9e15a5792b601ec03c702557cdb508184d6ce3fc255e2b\": container with ID starting with 767d814c712ed1734f9e15a5792b601ec03c702557cdb508184d6ce3fc255e2b not found: ID does not exist"
Nov 02 02:00:11.303110 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.303090    5639 scope.go:122] "RemoveContainer" containerID="919752363bc6a1ce089d0858dbc15cf073b144a55b23d3228ce1a781e74141e8"
Nov 02 02:00:11.321519 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.321494    5639 scope.go:122] "RemoveContainer" containerID="919752363bc6a1ce089d0858dbc15cf073b144a55b23d3228ce1a781e74141e8"
Nov 02 02:00:11.322066 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:11.322030    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"919752363bc6a1ce089d0858dbc15cf073b144a55b23d3228ce1a781e74141e8\": container with ID starting with 919752363bc6a1ce089d0858dbc15cf073b144a55b23d3228ce1a781e74141e8 not found: ID does not exist" containerID="919752363bc6a1ce089d0858dbc15cf073b144a55b23d3228ce1a781e74141e8"
Nov 02 02:00:11.322173 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.322086    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"919752363bc6a1ce089d0858dbc15cf073b144a55b23d3228ce1a781e74141e8"} err="failed to get container status \"919752363bc6a1ce089d0858dbc15cf073b144a55b23d3228ce1a781e74141e8\": rpc error: code = NotFound desc = could not find container \"919752363bc6a1ce089d0858dbc15cf073b144a55b23d3228ce1a781e74141e8\": container with ID starting with 919752363bc6a1ce089d0858dbc15cf073b144a55b23d3228ce1a781e74141e8 not found: ID does not exist"
Nov 02 02:00:11.577301 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.577104    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r"]
Nov 02 02:00:11.582802 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.582777    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-a7c06d-7bc7d44659-6wj5r"]
Nov 02 02:00:11.692418 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.692379    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf"]
Nov 02 02:00:11.696770 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.696749    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a7c06d-d9778485f-9qrvf"]
Nov 02 02:00:11.966588 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:11.966561    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5"
Nov 02 02:00:12.478993 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:12.478967    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-109b07-q2hk7"
Nov 02 02:00:12.546644 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:12.546616    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba22028e-a74a-4b4c-baef-dd9502007685-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba22028e-a74a-4b4c-baef-dd9502007685-maas-api-xks-ca\") pod \"ba22028e-a74a-4b4c-baef-dd9502007685\" (UID: \"ba22028e-a74a-4b4c-baef-dd9502007685\") "
Nov 02 02:00:12.546806 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:12.546789    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba22028e-a74a-4b4c-baef-dd9502007685-kube-api-access-6gcpp\" (UniqueName: \"kubernetes.io/projected/ba22028e-a74a-4b4c-baef-dd9502007685-kube-api-access-6gcpp\") pod \"ba22028e-a74a-4b4c-baef-dd9502007685\" (UID: \"ba22028e-a74a-4b4c-baef-dd9502007685\") "
Nov 02 02:00:12.546889 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:12.546875    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba22028e-a74a-4b4c-baef-dd9502007685-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba22028e-a74a-4b4c-baef-dd9502007685-maas-api-openshift-ca\") pod \"ba22028e-a74a-4b4c-baef-dd9502007685\" (UID: \"ba22028e-a74a-4b4c-baef-dd9502007685\") "
Nov 02 02:00:12.547438 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:12.547336    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba22028e-a74a-4b4c-baef-dd9502007685-maas-api-openshift-ca" pod "ba22028e-a74a-4b4c-baef-dd9502007685" (UID: "ba22028e-a74a-4b4c-baef-dd9502007685"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:12.547639 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:12.547616    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba22028e-a74a-4b4c-baef-dd9502007685-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:12.549705 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:12.549673    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba22028e-a74a-4b4c-baef-dd9502007685-maas-api-xks-ca" pod "ba22028e-a74a-4b4c-baef-dd9502007685" (UID: "ba22028e-a74a-4b4c-baef-dd9502007685"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:00:12.551076 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:12.550950    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba22028e-a74a-4b4c-baef-dd9502007685-kube-api-access-6gcpp" pod "ba22028e-a74a-4b4c-baef-dd9502007685" (UID: "ba22028e-a74a-4b4c-baef-dd9502007685"). InnerVolumeSpecName "kube-api-access-6gcpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:12.649656 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:12.649622    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba22028e-a74a-4b4c-baef-dd9502007685-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:12.649656 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:12.649660    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gcpp\" (UniqueName: \"kubernetes.io/projected/ba22028e-a74a-4b4c-baef-dd9502007685-kube-api-access-6gcpp\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:13.050322 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.044794    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk"
Nov 02 02:00:13.143653 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.143615    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c158b671-l8lwm"]
Nov 02 02:00:13.144970 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.144940    5639 cpu_manager.go:414] "containerMap: removing container" podUID="98bf5ee6-a486-456d-b9d8-d79bbaf5717e" containerName="payload-processing"
Nov 02 02:00:13.144970 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.144967    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98bf5ee6-a486-456d-b9d8-d79bbaf5717e" containerName="payload-processing"
Nov 02 02:00:13.145177 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.145012    5639 cpu_manager.go:414] "containerMap: removing container" podUID="ba22028e-a74a-4b4c-baef-dd9502007685" containerName="revoke-keys"
Nov 02 02:00:13.145177 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.145021    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba22028e-a74a-4b4c-baef-dd9502007685" containerName="revoke-keys"
Nov 02 02:00:13.145177 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.145086    5639 cpu_manager.go:414] "containerMap: removing container" podUID="1c1d68fb-24ac-454b-bf9c-02a894c43862" containerName="payload-pre-processing"
Nov 02 02:00:13.145177 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.145096    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c1d68fb-24ac-454b-bf9c-02a894c43862" containerName="payload-pre-processing"
Nov 02 02:00:13.145420 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.145341    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c1d68fb-24ac-454b-bf9c-02a894c43862" containerName="payload-pre-processing"
Nov 02 02:00:13.145420 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.145372    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="98bf5ee6-a486-456d-b9d8-d79bbaf5717e" containerName="payload-processing"
Nov 02 02:00:13.145420 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.145387    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba22028e-a74a-4b4c-baef-dd9502007685" containerName="revoke-keys"
Nov 02 02:00:13.172301 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.171573    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c158b671-l8lwm"
Nov 02 02:00:13.191222 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.191176    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c158b671-l8lwm"]
Nov 02 02:00:13.234060 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.234023    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c1d68fb-24ac-454b-bf9c-02a894c43862" path="/var/lib/kubelet/pods/1c1d68fb-24ac-454b-bf9c-02a894c43862/volumes"
Nov 02 02:00:13.234813 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.234791    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98bf5ee6-a486-456d-b9d8-d79bbaf5717e" path="/var/lib/kubelet/pods/98bf5ee6-a486-456d-b9d8-d79bbaf5717e/volumes"
Nov 02 02:00:13.292177 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.292134    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c471528a-da4f-46c6-b48d-3404eec749c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c158b671-l8lwm\" (UID: \"c471528a-da4f-46c6-b48d-3404eec749c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c158b671-l8lwm"
Nov 02 02:00:13.292411 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.292341    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spwtk\" (UniqueName: \"kubernetes.io/projected/c471528a-da4f-46c6-b48d-3404eec749c0-kube-api-access-spwtk\") pod \"maas-api-revoke-sub-c158b671-l8lwm\" (UID: \"c471528a-da4f-46c6-b48d-3404eec749c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c158b671-l8lwm"
Nov 02 02:00:13.292524 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.292504    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c471528a-da4f-46c6-b48d-3404eec749c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c158b671-l8lwm\" (UID: \"c471528a-da4f-46c6-b48d-3404eec749c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c158b671-l8lwm"
Nov 02 02:00:13.331741 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.331709    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-109b07-q2hk7"
Nov 02 02:00:13.331898 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.331732    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-109b07-q2hk7" event={"ID":"ba22028e-a74a-4b4c-baef-dd9502007685","Type":"ContainerDied","Data":"a17a8c4ff0ef797b70e2f5cccdd1196123ab1c423a528d710ef5a88ad382e32a"}
Nov 02 02:00:13.331898 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.331784    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a17a8c4ff0ef797b70e2f5cccdd1196123ab1c423a528d710ef5a88ad382e32a"
Nov 02 02:00:13.394022 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.393989    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c471528a-da4f-46c6-b48d-3404eec749c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c158b671-l8lwm\" (UID: \"c471528a-da4f-46c6-b48d-3404eec749c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c158b671-l8lwm"
Nov 02 02:00:13.394219 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.394058    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spwtk\" (UniqueName: \"kubernetes.io/projected/c471528a-da4f-46c6-b48d-3404eec749c0-kube-api-access-spwtk\") pod \"maas-api-revoke-sub-c158b671-l8lwm\" (UID: \"c471528a-da4f-46c6-b48d-3404eec749c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c158b671-l8lwm"
Nov 02 02:00:13.394219 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.394149    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c471528a-da4f-46c6-b48d-3404eec749c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c158b671-l8lwm\" (UID: \"c471528a-da4f-46c6-b48d-3404eec749c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c158b671-l8lwm"
Nov 02 02:00:13.394796 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.394771    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c471528a-da4f-46c6-b48d-3404eec749c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c158b671-l8lwm\" (UID: \"c471528a-da4f-46c6-b48d-3404eec749c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c158b671-l8lwm"
Nov 02 02:00:13.396715 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.396693    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c471528a-da4f-46c6-b48d-3404eec749c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c158b671-l8lwm\" (UID: \"c471528a-da4f-46c6-b48d-3404eec749c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c158b671-l8lwm"
Nov 02 02:00:13.455327 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.455269    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spwtk\" (UniqueName: \"kubernetes.io/projected/c471528a-da4f-46c6-b48d-3404eec749c0-kube-api-access-spwtk\") pod \"maas-api-revoke-sub-c158b671-l8lwm\" (UID: \"c471528a-da4f-46c6-b48d-3404eec749c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c158b671-l8lwm"
Nov 02 02:00:13.510550 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.510521    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c158b671-l8lwm"
Nov 02 02:00:13.992505 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:13.988371    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c158b671-l8lwm"]
Nov 02 02:00:14.341875 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:14.341838    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c158b671-l8lwm" event={"ID":"c471528a-da4f-46c6-b48d-3404eec749c0","Type":"ContainerStarted","Data":"e4a06c6ff694e8d3220fc992ea5cfeb98196063b456d942ec18cc5a1e071f0f8"}
Nov 02 02:00:14.342050 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:14.341904    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c158b671-l8lwm" event={"ID":"c471528a-da4f-46c6-b48d-3404eec749c0","Type":"ContainerStarted","Data":"55e983adbe6c97a8adf800fd42012c71d97a6bc56643b99dc39257dde3ebd51c"}
Nov 02 02:00:14.460244 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:14.460149    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c158b671-l8lwm" podStartSLOduration=1.460135052 podStartE2EDuration="1.460135052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:14.456259148 +0000 UTC m=+2079.359941253" watchObservedRunningTime="2026-11-02 02:00:14.460135052 +0000 UTC m=+2079.363817164"
Nov 02 02:00:15.221682 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:15.221654    5639 scope.go:122] "RemoveContainer" containerID="f85a792f91661b37a0e2dd2d097d52c0f6fa50841b9a2395fc020314e0ec01c2"
Nov 02 02:00:15.351804 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:15.351735    5639 generic.go:356] "Generic (PLEG): container finished" podID="c471528a-da4f-46c6-b48d-3404eec749c0" containerID="e4a06c6ff694e8d3220fc992ea5cfeb98196063b456d942ec18cc5a1e071f0f8" exitCode=0
Nov 02 02:00:15.351940 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:15.351817    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c158b671-l8lwm" event={"ID":"c471528a-da4f-46c6-b48d-3404eec749c0","Type":"ContainerDied","Data":"e4a06c6ff694e8d3220fc992ea5cfeb98196063b456d942ec18cc5a1e071f0f8"}
Nov 02 02:00:16.211799 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:16.211768    5639 scope.go:122] "RemoveContainer" containerID="976ac0649baf50eb48a1d2c1011b46799f1ce7704a2fe9aecc96c64050f9d5c4"
Nov 02 02:00:16.211956 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:16.211924    5639 scope.go:122] "RemoveContainer" containerID="649f0490dc7a01499109949b05f9d51a79dea32c61bb077ae54ca3e929cd7eb6"
Nov 02 02:00:16.212049 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:16.212027    5639 scope.go:122] "RemoveContainer" containerID="ed5d18c8969920ea383e38aff0a0dd2f9dcc1854ede9ee64e9085c3dd9a75c92"
Nov 02 02:00:16.212130 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:16.212099    5639 scope.go:122] "RemoveContainer" containerID="968beddf34969d722a12c637c77c54715305304ce161a683713e191a2ebbc92b"
Nov 02 02:00:16.364616 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:16.364587    5639 generic.go:356] "Generic (PLEG): container finished" podID="b669073a-3244-4c17-9255-197abd78c544" containerID="fe24512c60cbe43a1f55498e339e8bbba2ad102561eccae7b187b8e4aa570c18" exitCode=60
Nov 02 02:00:16.364970 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:16.364621    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498" event={"ID":"b669073a-3244-4c17-9255-197abd78c544","Type":"ContainerDied","Data":"fe24512c60cbe43a1f55498e339e8bbba2ad102561eccae7b187b8e4aa570c18"}
Nov 02 02:00:16.364970 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:16.364686    5639 scope.go:122] "RemoveContainer" containerID="f85a792f91661b37a0e2dd2d097d52c0f6fa50841b9a2395fc020314e0ec01c2"
Nov 02 02:00:16.365155 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:16.365097    5639 scope.go:122] "RemoveContainer" containerID="fe24512c60cbe43a1f55498e339e8bbba2ad102561eccae7b187b8e4aa570c18"
Nov 02 02:00:16.365571 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:16.365550    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498_odh-ai-gateway-infra(b669073a-3244-4c17-9255-197abd78c544)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498" podUID="b669073a-3244-4c17-9255-197abd78c544"
Nov 02 02:00:16.787397 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:16.787374    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c158b671-l8lwm"
Nov 02 02:00:16.876151 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:16.876123    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c471528a-da4f-46c6-b48d-3404eec749c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c471528a-da4f-46c6-b48d-3404eec749c0-maas-api-xks-ca\") pod \"c471528a-da4f-46c6-b48d-3404eec749c0\" (UID: \"c471528a-da4f-46c6-b48d-3404eec749c0\") "
Nov 02 02:00:16.876394 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:16.876375    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c471528a-da4f-46c6-b48d-3404eec749c0-kube-api-access-spwtk\" (UniqueName: \"kubernetes.io/projected/c471528a-da4f-46c6-b48d-3404eec749c0-kube-api-access-spwtk\") pod \"c471528a-da4f-46c6-b48d-3404eec749c0\" (UID: \"c471528a-da4f-46c6-b48d-3404eec749c0\") "
Nov 02 02:00:16.876452 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:16.876436    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c471528a-da4f-46c6-b48d-3404eec749c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c471528a-da4f-46c6-b48d-3404eec749c0-maas-api-openshift-ca\") pod \"c471528a-da4f-46c6-b48d-3404eec749c0\" (UID: \"c471528a-da4f-46c6-b48d-3404eec749c0\") "
Nov 02 02:00:16.876835 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:16.876772    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c471528a-da4f-46c6-b48d-3404eec749c0-maas-api-openshift-ca" pod "c471528a-da4f-46c6-b48d-3404eec749c0" (UID: "c471528a-da4f-46c6-b48d-3404eec749c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:16.877110 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:16.877090    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c471528a-da4f-46c6-b48d-3404eec749c0-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:16.878135 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:16.878113    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c471528a-da4f-46c6-b48d-3404eec749c0-maas-api-xks-ca" pod "c471528a-da4f-46c6-b48d-3404eec749c0" (UID: "c471528a-da4f-46c6-b48d-3404eec749c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:00:16.878682 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:16.878661    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c471528a-da4f-46c6-b48d-3404eec749c0-kube-api-access-spwtk" pod "c471528a-da4f-46c6-b48d-3404eec749c0" (UID: "c471528a-da4f-46c6-b48d-3404eec749c0"). InnerVolumeSpecName "kube-api-access-spwtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:16.978166 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:16.978136    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spwtk\" (UniqueName: \"kubernetes.io/projected/c471528a-da4f-46c6-b48d-3404eec749c0-kube-api-access-spwtk\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:16.978166 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:16.978164    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c471528a-da4f-46c6-b48d-3404eec749c0-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:17.212085 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:17.212056    5639 scope.go:122] "RemoveContainer" containerID="ab7123e22359bc51f700ef735c6816bf90e6653b8ef9b056061cdddf5aecfe1b"
Nov 02 02:00:17.212271 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:17.212247    5639 scope.go:122] "RemoveContainer" containerID="cebf987c48fe0c564b4c50bef01a88d6fc5591ff76589b4ead019b4e74c85a6a"
Nov 02 02:00:17.212445 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:17.212424    5639 scope.go:122] "RemoveContainer" containerID="27d514eb8a00e165dfce43f020cd0244297768ccbf1d8f80591501135b33bb5b"
Nov 02 02:00:17.212585 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:17.212561    5639 scope.go:122] "RemoveContainer" containerID="9d95ede9faf2e2a5de00dad4e9a0120c11cf22a215e0fd5ee32f8e00e973851f"
Nov 02 02:00:17.392126 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:17.392099    5639 generic.go:356] "Generic (PLEG): container finished" podID="5dabb512-1926-413b-87ab-2a9c87e685f7" containerID="05b47780751b5d78b23bb432a96493a945a8cc358cf29371440b455e40fcab91" exitCode=60
Nov 02 02:00:17.392535 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:17.392164    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7" event={"ID":"5dabb512-1926-413b-87ab-2a9c87e685f7","Type":"ContainerDied","Data":"05b47780751b5d78b23bb432a96493a945a8cc358cf29371440b455e40fcab91"}
Nov 02 02:00:17.392535 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:17.392234    5639 scope.go:122] "RemoveContainer" containerID="649f0490dc7a01499109949b05f9d51a79dea32c61bb077ae54ca3e929cd7eb6"
Nov 02 02:00:17.392672 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:17.392651    5639 scope.go:122] "RemoveContainer" containerID="05b47780751b5d78b23bb432a96493a945a8cc358cf29371440b455e40fcab91"
Nov 02 02:00:17.393063 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:17.393036    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7_odh-ai-gateway-infra(5dabb512-1926-413b-87ab-2a9c87e685f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7" podUID="5dabb512-1926-413b-87ab-2a9c87e685f7"
Nov 02 02:00:17.394191 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:17.394165    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c158b671-l8lwm" event={"ID":"c471528a-da4f-46c6-b48d-3404eec749c0","Type":"ContainerDied","Data":"55e983adbe6c97a8adf800fd42012c71d97a6bc56643b99dc39257dde3ebd51c"}
Nov 02 02:00:17.394541 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:17.394209    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55e983adbe6c97a8adf800fd42012c71d97a6bc56643b99dc39257dde3ebd51c"
Nov 02 02:00:17.394541 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:17.394177    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c158b671-l8lwm"
Nov 02 02:00:17.396642 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:17.396620    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm" event={"ID":"3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e","Type":"ContainerStarted","Data":"157b0b42b2742726f04f7112723acc1da6a45ea18362585e4eab028c487463c1"}
Nov 02 02:00:17.398660 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:17.398638    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z" event={"ID":"113179bd-c2bc-4fe6-909e-e1387c459bb9","Type":"ContainerStarted","Data":"32b62b1853fc932c3acab662f00039ccf1759f5c9de6bdc1160c4e3fc30da1dd"}
Nov 02 02:00:17.400899 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:17.400877    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc" event={"ID":"3fa637e0-ccb8-4713-a16b-f4c353d3cb8f","Type":"ContainerStarted","Data":"bc171aeae1b910d7d0cd1a1c0f1d0f2d9dcae0d41ec7e4566dbbd65b78fef34a"}
Nov 02 02:00:17.401167 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:17.401155    5639 scope.go:122] "RemoveContainer" containerID="bc171aeae1b910d7d0cd1a1c0f1d0f2d9dcae0d41ec7e4566dbbd65b78fef34a"
Nov 02 02:00:17.401423 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:17.401405    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc_odh-ai-gateway-infra(3fa637e0-ccb8-4713-a16b-f4c353d3cb8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc" podUID="3fa637e0-ccb8-4713-a16b-f4c353d3cb8f"
Nov 02 02:00:17.491770 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:17.491744    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498"]
Nov 02 02:00:17.956260 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:17.945238    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:00:18.205515 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.205487    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498"
Nov 02 02:00:18.211907 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.211884    5639 scope.go:122] "RemoveContainer" containerID="edcf164a962f0f56673dea4e446d47f40052f6e84f9177571b151e1669e96672"
Nov 02 02:00:18.212050 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.212032    5639 scope.go:122] "RemoveContainer" containerID="462817b959ded5b42e207de08949a580dac6b4e5bc5904469bfb66fa8a37eaa6"
Nov 02 02:00:18.325742 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.325707    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b669073a-3244-4c17-9255-197abd78c544-kube-api-access-qqdcz\" (UniqueName: \"kubernetes.io/projected/b669073a-3244-4c17-9255-197abd78c544-kube-api-access-qqdcz\") pod \"b669073a-3244-4c17-9255-197abd78c544\" (UID: \"b669073a-3244-4c17-9255-197abd78c544\") "
Nov 02 02:00:18.325925 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.325902    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b669073a-3244-4c17-9255-197abd78c544-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b669073a-3244-4c17-9255-197abd78c544-maas-api-service-ca\") pod \"b669073a-3244-4c17-9255-197abd78c544\" (UID: \"b669073a-3244-4c17-9255-197abd78c544\") "
Nov 02 02:00:18.326643 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.326543    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b669073a-3244-4c17-9255-197abd78c544-maas-api-service-ca" pod "b669073a-3244-4c17-9255-197abd78c544" (UID: "b669073a-3244-4c17-9255-197abd78c544"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:18.328483 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.327333    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b669073a-3244-4c17-9255-197abd78c544-maas-api-service-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:18.378386 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.370400    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b669073a-3244-4c17-9255-197abd78c544-kube-api-access-qqdcz" pod "b669073a-3244-4c17-9255-197abd78c544" (UID: "b669073a-3244-4c17-9255-197abd78c544"). InnerVolumeSpecName "kube-api-access-qqdcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:18.412217 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.412181    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9" event={"ID":"0720184c-4f19-41d9-871f-b3a208201818","Type":"ContainerStarted","Data":"eacb1ebd03cc8564aec55cb6d28edff740e4ebfdf940f0561934ed9f0ffc67aa"}
Nov 02 02:00:18.414209 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.414156    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498"
Nov 02 02:00:18.414209 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.414185    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498" event={"ID":"b669073a-3244-4c17-9255-197abd78c544","Type":"ContainerDied","Data":"c2c3c7799e04d778a4c57e5aa48ca58770e8aea6800b1ec26e8b0968a5056e2c"}
Nov 02 02:00:18.414490 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.414241    5639 scope.go:122] "RemoveContainer" containerID="fe24512c60cbe43a1f55498e339e8bbba2ad102561eccae7b187b8e4aa570c18"
Nov 02 02:00:18.416697 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.416668    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2" event={"ID":"2c0d592c-c444-4131-ac9c-13e3980e5a6d","Type":"ContainerStarted","Data":"93e7ab285246d327289ada398e2fa8d8f3928ac37eb6de9ddfe0a7a60cb42dcf"}
Nov 02 02:00:18.421906 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.421872    5639 generic.go:356] "Generic (PLEG): container finished" podID="3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e" containerID="157b0b42b2742726f04f7112723acc1da6a45ea18362585e4eab028c487463c1" exitCode=60
Nov 02 02:00:18.422032 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.421942    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm" event={"ID":"3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e","Type":"ContainerDied","Data":"157b0b42b2742726f04f7112723acc1da6a45ea18362585e4eab028c487463c1"}
Nov 02 02:00:18.422468 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.422448    5639 scope.go:122] "RemoveContainer" containerID="157b0b42b2742726f04f7112723acc1da6a45ea18362585e4eab028c487463c1"
Nov 02 02:00:18.422853 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:18.422832    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm_odh-ai-gateway-infra(3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm" podUID="3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e"
Nov 02 02:00:18.424656 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.424631    5639 generic.go:356] "Generic (PLEG): container finished" podID="113179bd-c2bc-4fe6-909e-e1387c459bb9" containerID="32b62b1853fc932c3acab662f00039ccf1759f5c9de6bdc1160c4e3fc30da1dd" exitCode=60
Nov 02 02:00:18.424763 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.424711    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z" event={"ID":"113179bd-c2bc-4fe6-909e-e1387c459bb9","Type":"ContainerDied","Data":"32b62b1853fc932c3acab662f00039ccf1759f5c9de6bdc1160c4e3fc30da1dd"}
Nov 02 02:00:18.425174 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.425118    5639 scope.go:122] "RemoveContainer" containerID="32b62b1853fc932c3acab662f00039ccf1759f5c9de6bdc1160c4e3fc30da1dd"
Nov 02 02:00:18.425525 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:18.425504    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z_odh-ai-gateway-infra(113179bd-c2bc-4fe6-909e-e1387c459bb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z" podUID="113179bd-c2bc-4fe6-909e-e1387c459bb9"
Nov 02 02:00:18.426943 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.426925    5639 generic.go:356] "Generic (PLEG): container finished" podID="3fa637e0-ccb8-4713-a16b-f4c353d3cb8f" containerID="bc171aeae1b910d7d0cd1a1c0f1d0f2d9dcae0d41ec7e4566dbbd65b78fef34a" exitCode=60
Nov 02 02:00:18.427040 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.426988    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc" event={"ID":"3fa637e0-ccb8-4713-a16b-f4c353d3cb8f","Type":"ContainerDied","Data":"bc171aeae1b910d7d0cd1a1c0f1d0f2d9dcae0d41ec7e4566dbbd65b78fef34a"}
Nov 02 02:00:18.427370 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.427348    5639 scope.go:122] "RemoveContainer" containerID="bc171aeae1b910d7d0cd1a1c0f1d0f2d9dcae0d41ec7e4566dbbd65b78fef34a"
Nov 02 02:00:18.427719 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:18.427695    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc_odh-ai-gateway-infra(3fa637e0-ccb8-4713-a16b-f4c353d3cb8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc" podUID="3fa637e0-ccb8-4713-a16b-f4c353d3cb8f"
Nov 02 02:00:18.428976 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.428768    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqdcz\" (UniqueName: \"kubernetes.io/projected/b669073a-3244-4c17-9255-197abd78c544-kube-api-access-qqdcz\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:18.429151 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.429020    5639 scope.go:122] "RemoveContainer" containerID="968beddf34969d722a12c637c77c54715305304ce161a683713e191a2ebbc92b"
Nov 02 02:00:18.429596 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.429568    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh" event={"ID":"d96d25d2-704b-40d5-9a6b-116c4eb3a4a4","Type":"ContainerStarted","Data":"13196c24e14ca88c7d9ba1d90f059ecfe4b8f720376e6e1f2338d363225d7ae3"}
Nov 02 02:00:18.431783 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.431765    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt" event={"ID":"5184916a-2d9b-40cc-9872-9fb0cd055b41","Type":"ContainerStarted","Data":"5881dfa968046050b057b7f42a8e3c985ef196e08b70cf4feb788f8c91659264"}
Nov 02 02:00:18.446440 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.446404    5639 scope.go:122] "RemoveContainer" containerID="ed5d18c8969920ea383e38aff0a0dd2f9dcc1854ede9ee64e9085c3dd9a75c92"
Nov 02 02:00:18.470733 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.468502    5639 scope.go:122] "RemoveContainer" containerID="976ac0649baf50eb48a1d2c1011b46799f1ce7704a2fe9aecc96c64050f9d5c4"
Nov 02 02:00:18.520971 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.520879    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7"]
Nov 02 02:00:18.643930 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.643892    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc"]
Nov 02 02:00:18.757162 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.757115    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z"]
Nov 02 02:00:18.794108 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.794075    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7"
Nov 02 02:00:18.939617 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.939579    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dabb512-1926-413b-87ab-2a9c87e685f7-kube-api-access-kdv2m\" (UniqueName: \"kubernetes.io/projected/5dabb512-1926-413b-87ab-2a9c87e685f7-kube-api-access-kdv2m\") pod \"5dabb512-1926-413b-87ab-2a9c87e685f7\" (UID: \"5dabb512-1926-413b-87ab-2a9c87e685f7\") "
Nov 02 02:00:18.939835 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.939816    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dabb512-1926-413b-87ab-2a9c87e685f7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dabb512-1926-413b-87ab-2a9c87e685f7-maas-api-service-ca\") pod \"5dabb512-1926-413b-87ab-2a9c87e685f7\" (UID: \"5dabb512-1926-413b-87ab-2a9c87e685f7\") "
Nov 02 02:00:18.940142 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.940121    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dabb512-1926-413b-87ab-2a9c87e685f7-maas-api-service-ca" pod "5dabb512-1926-413b-87ab-2a9c87e685f7" (UID: "5dabb512-1926-413b-87ab-2a9c87e685f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:18.940696 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.940676    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dabb512-1926-413b-87ab-2a9c87e685f7-maas-api-service-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:18.961192 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.961152    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dabb512-1926-413b-87ab-2a9c87e685f7-kube-api-access-kdv2m" pod "5dabb512-1926-413b-87ab-2a9c87e685f7" (UID: "5dabb512-1926-413b-87ab-2a9c87e685f7"). InnerVolumeSpecName "kube-api-access-kdv2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:18.981604 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.981580    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm"]
Nov 02 02:00:18.996820 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:18.996795    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498"]
Nov 02 02:00:19.008785 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.008746    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893080-tx498"]
Nov 02 02:00:19.042838 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.042762    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdv2m\" (UniqueName: \"kubernetes.io/projected/5dabb512-1926-413b-87ab-2a9c87e685f7-kube-api-access-kdv2m\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:19.351738 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.351702    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b669073a-3244-4c17-9255-197abd78c544" path="/var/lib/kubelet/pods/b669073a-3244-4c17-9255-197abd78c544/volumes"
Nov 02 02:00:19.401880 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.399077    5639 scope.go:122] "RemoveContainer" containerID="741979e020bee66ec59e9d4c438e8ea0462f013ae6183547a9aca528fda88980"
Nov 02 02:00:19.442008 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.441975    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw" event={"ID":"3f5659fe-839b-4498-b9eb-2b76d6631417","Type":"ContainerStarted","Data":"09c43c25ca5d0e1da8fbb8d771535a11b18126f1d19ab8086ab36f57f02fc4db"}
Nov 02 02:00:19.444016 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.443990    5639 generic.go:356] "Generic (PLEG): container finished" podID="0720184c-4f19-41d9-871f-b3a208201818" containerID="eacb1ebd03cc8564aec55cb6d28edff740e4ebfdf940f0561934ed9f0ffc67aa" exitCode=60
Nov 02 02:00:19.444143 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.444063    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9" event={"ID":"0720184c-4f19-41d9-871f-b3a208201818","Type":"ContainerDied","Data":"eacb1ebd03cc8564aec55cb6d28edff740e4ebfdf940f0561934ed9f0ffc67aa"}
Nov 02 02:00:19.444143 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.444123    5639 scope.go:122] "RemoveContainer" containerID="27d514eb8a00e165dfce43f020cd0244297768ccbf1d8f80591501135b33bb5b"
Nov 02 02:00:19.444623 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.444603    5639 scope.go:122] "RemoveContainer" containerID="eacb1ebd03cc8564aec55cb6d28edff740e4ebfdf940f0561934ed9f0ffc67aa"
Nov 02 02:00:19.445006 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:19.444969    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9_odh-ai-gateway-infra(0720184c-4f19-41d9-871f-b3a208201818)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9" podUID="0720184c-4f19-41d9-871f-b3a208201818"
Nov 02 02:00:19.447630 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.447610    5639 generic.go:356] "Generic (PLEG): container finished" podID="2c0d592c-c444-4131-ac9c-13e3980e5a6d" containerID="93e7ab285246d327289ada398e2fa8d8f3928ac37eb6de9ddfe0a7a60cb42dcf" exitCode=60
Nov 02 02:00:19.447747 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.447672    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2" event={"ID":"2c0d592c-c444-4131-ac9c-13e3980e5a6d","Type":"ContainerDied","Data":"93e7ab285246d327289ada398e2fa8d8f3928ac37eb6de9ddfe0a7a60cb42dcf"}
Nov 02 02:00:19.448002 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.447986    5639 scope.go:122] "RemoveContainer" containerID="93e7ab285246d327289ada398e2fa8d8f3928ac37eb6de9ddfe0a7a60cb42dcf"
Nov 02 02:00:19.448345 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:19.448320    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2_odh-ai-gateway-infra(2c0d592c-c444-4131-ac9c-13e3980e5a6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2" podUID="2c0d592c-c444-4131-ac9c-13e3980e5a6d"
Nov 02 02:00:19.449828 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.449807    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps" event={"ID":"e5b031ba-6e9e-4852-abb4-218ab35f22b9","Type":"ContainerStarted","Data":"db5c76c924c17709888f4d0cd5876bc8757fe116ac29792923b57ce1ebc1b0a7"}
Nov 02 02:00:19.451476 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.451456    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7"
Nov 02 02:00:19.451574 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.451483    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7" event={"ID":"5dabb512-1926-413b-87ab-2a9c87e685f7","Type":"ContainerDied","Data":"31226ed0840322d4ef0e44fa23c5df5c0b4750ba0a9b1bc8093a14857aae0d6c"}
Nov 02 02:00:19.459057 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.459037    5639 generic.go:356] "Generic (PLEG): container finished" podID="d96d25d2-704b-40d5-9a6b-116c4eb3a4a4" containerID="13196c24e14ca88c7d9ba1d90f059ecfe4b8f720376e6e1f2338d363225d7ae3" exitCode=60
Nov 02 02:00:19.459146 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.459110    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh" event={"ID":"d96d25d2-704b-40d5-9a6b-116c4eb3a4a4","Type":"ContainerDied","Data":"13196c24e14ca88c7d9ba1d90f059ecfe4b8f720376e6e1f2338d363225d7ae3"}
Nov 02 02:00:19.459515 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.459496    5639 scope.go:122] "RemoveContainer" containerID="13196c24e14ca88c7d9ba1d90f059ecfe4b8f720376e6e1f2338d363225d7ae3"
Nov 02 02:00:19.459971 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:19.459944    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh_odh-ai-gateway-infra(d96d25d2-704b-40d5-9a6b-116c4eb3a4a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh" podUID="d96d25d2-704b-40d5-9a6b-116c4eb3a4a4"
Nov 02 02:00:19.461373 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.461353    5639 generic.go:356] "Generic (PLEG): container finished" podID="5184916a-2d9b-40cc-9872-9fb0cd055b41" containerID="5881dfa968046050b057b7f42a8e3c985ef196e08b70cf4feb788f8c91659264" exitCode=60
Nov 02 02:00:19.461459 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.461429    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt" event={"ID":"5184916a-2d9b-40cc-9872-9fb0cd055b41","Type":"ContainerDied","Data":"5881dfa968046050b057b7f42a8e3c985ef196e08b70cf4feb788f8c91659264"}
Nov 02 02:00:19.461753 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.461736    5639 scope.go:122] "RemoveContainer" containerID="5881dfa968046050b057b7f42a8e3c985ef196e08b70cf4feb788f8c91659264"
Nov 02 02:00:19.462049 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:19.461994    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt_odh-ai-gateway-infra(5184916a-2d9b-40cc-9872-9fb0cd055b41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt" podUID="5184916a-2d9b-40cc-9872-9fb0cd055b41"
Nov 02 02:00:19.474178 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.474156    5639 scope.go:122] "RemoveContainer" containerID="cebf987c48fe0c564b4c50bef01a88d6fc5591ff76589b4ead019b4e74c85a6a"
Nov 02 02:00:19.499975 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.499949    5639 scope.go:122] "RemoveContainer" containerID="05b47780751b5d78b23bb432a96493a945a8cc358cf29371440b455e40fcab91"
Nov 02 02:00:19.517826 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.517470    5639 scope.go:122] "RemoveContainer" containerID="ab7123e22359bc51f700ef735c6816bf90e6653b8ef9b056061cdddf5aecfe1b"
Nov 02 02:00:19.741706 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.741619    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9"]
Nov 02 02:00:19.791069 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.791035    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt"]
Nov 02 02:00:19.806537 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.806512    5639 scope.go:122] "RemoveContainer" containerID="9d95ede9faf2e2a5de00dad4e9a0120c11cf22a215e0fd5ee32f8e00e973851f"
Nov 02 02:00:19.851030 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.851009    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm"
Nov 02 02:00:19.899127 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.899095    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh"]
Nov 02 02:00:19.968476 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.968447    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z"
Nov 02 02:00:19.975520 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:19.975424    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc"
Nov 02 02:00:20.044025 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.043996    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa637e0-ccb8-4713-a16b-f4c353d3cb8f-kube-api-access-xvfsp\" (UniqueName: \"kubernetes.io/projected/3fa637e0-ccb8-4713-a16b-f4c353d3cb8f-kube-api-access-xvfsp\") pod \"3fa637e0-ccb8-4713-a16b-f4c353d3cb8f\" (UID: \"3fa637e0-ccb8-4713-a16b-f4c353d3cb8f\") "
Nov 02 02:00:20.044191 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.044156    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa637e0-ccb8-4713-a16b-f4c353d3cb8f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa637e0-ccb8-4713-a16b-f4c353d3cb8f-maas-api-service-ca\") pod \"3fa637e0-ccb8-4713-a16b-f4c353d3cb8f\" (UID: \"3fa637e0-ccb8-4713-a16b-f4c353d3cb8f\") "
Nov 02 02:00:20.044312 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.044228    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/113179bd-c2bc-4fe6-909e-e1387c459bb9-kube-api-access-glk7f\" (UniqueName: \"kubernetes.io/projected/113179bd-c2bc-4fe6-909e-e1387c459bb9-kube-api-access-glk7f\") pod \"113179bd-c2bc-4fe6-909e-e1387c459bb9\" (UID: \"113179bd-c2bc-4fe6-909e-e1387c459bb9\") "
Nov 02 02:00:20.044560 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.044505    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e-maas-api-service-ca\") pod \"3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e\" (UID: \"3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e\") "
Nov 02 02:00:20.044726 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.044592    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e-kube-api-access-gtjsm\" (UniqueName: \"kubernetes.io/projected/3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e-kube-api-access-gtjsm\") pod \"3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e\" (UID: \"3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e\") "
Nov 02 02:00:20.044726 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.044700    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/113179bd-c2bc-4fe6-909e-e1387c459bb9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/113179bd-c2bc-4fe6-909e-e1387c459bb9-maas-api-service-ca\") pod \"113179bd-c2bc-4fe6-909e-e1387c459bb9\" (UID: \"113179bd-c2bc-4fe6-909e-e1387c459bb9\") "
Nov 02 02:00:20.045624 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.045426    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa637e0-ccb8-4713-a16b-f4c353d3cb8f-maas-api-service-ca" pod "3fa637e0-ccb8-4713-a16b-f4c353d3cb8f" (UID: "3fa637e0-ccb8-4713-a16b-f4c353d3cb8f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:20.045732 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.045702    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e-maas-api-service-ca" pod "3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e" (UID: "3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:20.046194 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.046143    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa637e0-ccb8-4713-a16b-f4c353d3cb8f-maas-api-service-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:20.046194 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.046170    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e-maas-api-service-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:20.046392 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.046349    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/113179bd-c2bc-4fe6-909e-e1387c459bb9-maas-api-service-ca" pod "113179bd-c2bc-4fe6-909e-e1387c459bb9" (UID: "113179bd-c2bc-4fe6-909e-e1387c459bb9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:20.046987 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.046966    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa637e0-ccb8-4713-a16b-f4c353d3cb8f-kube-api-access-xvfsp" pod "3fa637e0-ccb8-4713-a16b-f4c353d3cb8f" (UID: "3fa637e0-ccb8-4713-a16b-f4c353d3cb8f"). InnerVolumeSpecName "kube-api-access-xvfsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:20.048345 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.048321    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e-kube-api-access-gtjsm" pod "3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e" (UID: "3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e"). InnerVolumeSpecName "kube-api-access-gtjsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:20.049312 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.049293    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113179bd-c2bc-4fe6-909e-e1387c459bb9-kube-api-access-glk7f" pod "113179bd-c2bc-4fe6-909e-e1387c459bb9" (UID: "113179bd-c2bc-4fe6-909e-e1387c459bb9"). InnerVolumeSpecName "kube-api-access-glk7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:20.147570 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.147526    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/113179bd-c2bc-4fe6-909e-e1387c459bb9-maas-api-service-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:20.147570 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.147559    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvfsp\" (UniqueName: \"kubernetes.io/projected/3fa637e0-ccb8-4713-a16b-f4c353d3cb8f-kube-api-access-xvfsp\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:20.147570 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.147575    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glk7f\" (UniqueName: \"kubernetes.io/projected/113179bd-c2bc-4fe6-909e-e1387c459bb9-kube-api-access-glk7f\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:20.147824 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.147588    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtjsm\" (UniqueName: \"kubernetes.io/projected/3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e-kube-api-access-gtjsm\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:20.413500 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.413393    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2"]
Nov 02 02:00:20.525173 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.525081    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc" event={"ID":"2aefa8c8-6056-4fd0-8e70-739dbd421750","Type":"ContainerStarted","Data":"35be26f9da8fb5c358c9e2212b04d50dc4a6797d4020c77b6bffcd9ff066fea8"}
Nov 02 02:00:20.528848 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.528825    5639 generic.go:356] "Generic (PLEG): container finished" podID="e5b031ba-6e9e-4852-abb4-218ab35f22b9" containerID="db5c76c924c17709888f4d0cd5876bc8757fe116ac29792923b57ce1ebc1b0a7" exitCode=60
Nov 02 02:00:20.528944 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.528900    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps" event={"ID":"e5b031ba-6e9e-4852-abb4-218ab35f22b9","Type":"ContainerDied","Data":"db5c76c924c17709888f4d0cd5876bc8757fe116ac29792923b57ce1ebc1b0a7"}
Nov 02 02:00:20.528996 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.528971    5639 scope.go:122] "RemoveContainer" containerID="edcf164a962f0f56673dea4e446d47f40052f6e84f9177571b151e1669e96672"
Nov 02 02:00:20.529517 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.529495    5639 scope.go:122] "RemoveContainer" containerID="db5c76c924c17709888f4d0cd5876bc8757fe116ac29792923b57ce1ebc1b0a7"
Nov 02 02:00:20.529970 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:20.529946    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps_odh-ai-gateway-infra(e5b031ba-6e9e-4852-abb4-218ab35f22b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps" podUID="e5b031ba-6e9e-4852-abb4-218ab35f22b9"
Nov 02 02:00:20.531786 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.531762    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm" event={"ID":"3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e","Type":"ContainerDied","Data":"11aee98cb5361b121615c74492d172988991fdf0e51c40dd84a83c122296a45f"}
Nov 02 02:00:20.531786 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.531779    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm"
Nov 02 02:00:20.536070 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.536049    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z"
Nov 02 02:00:20.536169 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.536102    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z" event={"ID":"113179bd-c2bc-4fe6-909e-e1387c459bb9","Type":"ContainerDied","Data":"78e703413f17628e5048c0199784e641e7e2d3f69df70e5c21cdfb6bd76c94b0"}
Nov 02 02:00:20.540344 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.537650    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc" event={"ID":"3fa637e0-ccb8-4713-a16b-f4c353d3cb8f","Type":"ContainerDied","Data":"68cdefd866b12df0e366a7af869a29455b1462a6f93e9fcc75520baa817044ca"}
Nov 02 02:00:20.540344 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.537741    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc"
Nov 02 02:00:20.548623 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.548591    5639 generic.go:356] "Generic (PLEG): container finished" podID="3f5659fe-839b-4498-b9eb-2b76d6631417" containerID="09c43c25ca5d0e1da8fbb8d771535a11b18126f1d19ab8086ab36f57f02fc4db" exitCode=60
Nov 02 02:00:20.548792 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.548722    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw" event={"ID":"3f5659fe-839b-4498-b9eb-2b76d6631417","Type":"ContainerDied","Data":"09c43c25ca5d0e1da8fbb8d771535a11b18126f1d19ab8086ab36f57f02fc4db"}
Nov 02 02:00:20.549463 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.549441    5639 scope.go:122] "RemoveContainer" containerID="09c43c25ca5d0e1da8fbb8d771535a11b18126f1d19ab8086ab36f57f02fc4db"
Nov 02 02:00:20.549928 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:20.549903    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw_odh-ai-gateway-infra(3f5659fe-839b-4498-b9eb-2b76d6631417)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw" podUID="3f5659fe-839b-4498-b9eb-2b76d6631417"
Nov 02 02:00:20.554371 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.554356    5639 scope.go:122] "RemoveContainer" containerID="157b0b42b2742726f04f7112723acc1da6a45ea18362585e4eab028c487463c1"
Nov 02 02:00:20.741039 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.740518    5639 scope.go:122] "RemoveContainer" containerID="32b62b1853fc932c3acab662f00039ccf1759f5c9de6bdc1160c4e3fc30da1dd"
Nov 02 02:00:20.832672 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.830539    5639 scope.go:122] "RemoveContainer" containerID="bc171aeae1b910d7d0cd1a1c0f1d0f2d9dcae0d41ec7e4566dbbd65b78fef34a"
Nov 02 02:00:20.858126 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.858101    5639 scope.go:122] "RemoveContainer" containerID="462817b959ded5b42e207de08949a580dac6b4e5bc5904469bfb66fa8a37eaa6"
Nov 02 02:00:20.928416 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.928339    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7"]
Nov 02 02:00:20.934925 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.932434    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893080-brgj7"]
Nov 02 02:00:20.957817 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:20.950892    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw"]
Nov 02 02:00:21.035775 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.035677    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2"
Nov 02 02:00:21.243261 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.243228    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c0d592c-c444-4131-ac9c-13e3980e5a6d-kube-api-access-8g5sp\" (UniqueName: \"kubernetes.io/projected/2c0d592c-c444-4131-ac9c-13e3980e5a6d-kube-api-access-8g5sp\") pod \"2c0d592c-c444-4131-ac9c-13e3980e5a6d\" (UID: \"2c0d592c-c444-4131-ac9c-13e3980e5a6d\") "
Nov 02 02:00:21.247152 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.247123    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c0d592c-c444-4131-ac9c-13e3980e5a6d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c0d592c-c444-4131-ac9c-13e3980e5a6d-maas-api-service-ca\") pod \"2c0d592c-c444-4131-ac9c-13e3980e5a6d\" (UID: \"2c0d592c-c444-4131-ac9c-13e3980e5a6d\") "
Nov 02 02:00:21.250546 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.250509    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c0d592c-c444-4131-ac9c-13e3980e5a6d-maas-api-service-ca" pod "2c0d592c-c444-4131-ac9c-13e3980e5a6d" (UID: "2c0d592c-c444-4131-ac9c-13e3980e5a6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:21.251864 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.251844    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c0d592c-c444-4131-ac9c-13e3980e5a6d-kube-api-access-8g5sp" pod "2c0d592c-c444-4131-ac9c-13e3980e5a6d" (UID: "2c0d592c-c444-4131-ac9c-13e3980e5a6d"). InnerVolumeSpecName "kube-api-access-8g5sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:21.328243 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.328175    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh"
Nov 02 02:00:21.350359 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.350322    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dabb512-1926-413b-87ab-2a9c87e685f7" path="/var/lib/kubelet/pods/5dabb512-1926-413b-87ab-2a9c87e685f7/volumes"
Nov 02 02:00:21.358458 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.351132    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d96d25d2-704b-40d5-9a6b-116c4eb3a4a4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d96d25d2-704b-40d5-9a6b-116c4eb3a4a4-maas-api-service-ca\") pod \"d96d25d2-704b-40d5-9a6b-116c4eb3a4a4\" (UID: \"d96d25d2-704b-40d5-9a6b-116c4eb3a4a4\") "
Nov 02 02:00:21.359212 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.359182    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm"]
Nov 02 02:00:21.359369 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.359233    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-39ceb2-29893080-hhvcm"]
Nov 02 02:00:21.359369 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.359259    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps"]
Nov 02 02:00:21.424538 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.424502    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d96d25d2-704b-40d5-9a6b-116c4eb3a4a4-maas-api-service-ca" pod "d96d25d2-704b-40d5-9a6b-116c4eb3a4a4" (UID: "d96d25d2-704b-40d5-9a6b-116c4eb3a4a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:21.424685 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.424562    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d96d25d2-704b-40d5-9a6b-116c4eb3a4a4-kube-api-access-5b8bd\" (UniqueName: \"kubernetes.io/projected/d96d25d2-704b-40d5-9a6b-116c4eb3a4a4-kube-api-access-5b8bd\") pod \"d96d25d2-704b-40d5-9a6b-116c4eb3a4a4\" (UID: \"d96d25d2-704b-40d5-9a6b-116c4eb3a4a4\") "
Nov 02 02:00:21.425398 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.425373    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g5sp\" (UniqueName: \"kubernetes.io/projected/2c0d592c-c444-4131-ac9c-13e3980e5a6d-kube-api-access-8g5sp\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:21.425398 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.425399    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d96d25d2-704b-40d5-9a6b-116c4eb3a4a4-maas-api-service-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:21.425519 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.425412    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c0d592c-c444-4131-ac9c-13e3980e5a6d-maas-api-service-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:21.430099 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.430069    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d96d25d2-704b-40d5-9a6b-116c4eb3a4a4-kube-api-access-5b8bd" pod "d96d25d2-704b-40d5-9a6b-116c4eb3a4a4" (UID: "d96d25d2-704b-40d5-9a6b-116c4eb3a4a4"). InnerVolumeSpecName "kube-api-access-5b8bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:21.521375 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.521343    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9"
Nov 02 02:00:21.523357 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.523341    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt"
Nov 02 02:00:21.526710 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.526689    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b8bd\" (UniqueName: \"kubernetes.io/projected/d96d25d2-704b-40d5-9a6b-116c4eb3a4a4-kube-api-access-5b8bd\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:21.613043 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.613003    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2" event={"ID":"2c0d592c-c444-4131-ac9c-13e3980e5a6d","Type":"ContainerDied","Data":"a575ffb6c0157d2e5a0f1bcd78d696eba798bf726aa7b30ae005c58843a80037"}
Nov 02 02:00:21.613224 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.613080    5639 scope.go:122] "RemoveContainer" containerID="93e7ab285246d327289ada398e2fa8d8f3928ac37eb6de9ddfe0a7a60cb42dcf"
Nov 02 02:00:21.613381 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.613362    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2"
Nov 02 02:00:21.621862 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.621833    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh" event={"ID":"d96d25d2-704b-40d5-9a6b-116c4eb3a4a4","Type":"ContainerDied","Data":"2d2ce0eb93a4940db1dc2aaf4bb980c207d073e41f2d441f911b2c2a36aa82c2"}
Nov 02 02:00:21.623241 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.623214    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt" event={"ID":"5184916a-2d9b-40cc-9872-9fb0cd055b41","Type":"ContainerDied","Data":"c501d6c2c1491cc54a611060df864366d1089b93cf67874f3c77dbf3a3c8099c"}
Nov 02 02:00:21.624626 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.624605    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh"
Nov 02 02:00:21.625238 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.625216    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt"
Nov 02 02:00:21.627596 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.627192    5639 scope.go:122] "RemoveContainer" containerID="13196c24e14ca88c7d9ba1d90f059ecfe4b8f720376e6e1f2338d363225d7ae3"
Nov 02 02:00:21.627596 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.627341    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0720184c-4f19-41d9-871f-b3a208201818-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0720184c-4f19-41d9-871f-b3a208201818-maas-api-service-ca\") pod \"0720184c-4f19-41d9-871f-b3a208201818\" (UID: \"0720184c-4f19-41d9-871f-b3a208201818\") "
Nov 02 02:00:21.627596 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.627394    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0720184c-4f19-41d9-871f-b3a208201818-kube-api-access-qqrcl\" (UniqueName: \"kubernetes.io/projected/0720184c-4f19-41d9-871f-b3a208201818-kube-api-access-qqrcl\") pod \"0720184c-4f19-41d9-871f-b3a208201818\" (UID: \"0720184c-4f19-41d9-871f-b3a208201818\") "
Nov 02 02:00:21.627596 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.627467    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5184916a-2d9b-40cc-9872-9fb0cd055b41-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5184916a-2d9b-40cc-9872-9fb0cd055b41-maas-api-service-ca\") pod \"5184916a-2d9b-40cc-9872-9fb0cd055b41\" (UID: \"5184916a-2d9b-40cc-9872-9fb0cd055b41\") "
Nov 02 02:00:21.627741 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.627710    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5184916a-2d9b-40cc-9872-9fb0cd055b41-kube-api-access-xqn5r\" (UniqueName: \"kubernetes.io/projected/5184916a-2d9b-40cc-9872-9fb0cd055b41-kube-api-access-xqn5r\") pod \"5184916a-2d9b-40cc-9872-9fb0cd055b41\" (UID: \"5184916a-2d9b-40cc-9872-9fb0cd055b41\") "
Nov 02 02:00:21.630572 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.630544    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9" event={"ID":"0720184c-4f19-41d9-871f-b3a208201818","Type":"ContainerDied","Data":"a62d998b79621bf69270062c8627829799a8db7625564875aea5a7231e212548"}
Nov 02 02:00:21.630665 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.630650    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9"
Nov 02 02:00:21.632527 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.631940    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0720184c-4f19-41d9-871f-b3a208201818-maas-api-service-ca" pod "0720184c-4f19-41d9-871f-b3a208201818" (UID: "0720184c-4f19-41d9-871f-b3a208201818"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:21.632527 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.632415    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5184916a-2d9b-40cc-9872-9fb0cd055b41-maas-api-service-ca" pod "5184916a-2d9b-40cc-9872-9fb0cd055b41" (UID: "5184916a-2d9b-40cc-9872-9fb0cd055b41"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:21.633403 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.633379    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5184916a-2d9b-40cc-9872-9fb0cd055b41-kube-api-access-xqn5r" pod "5184916a-2d9b-40cc-9872-9fb0cd055b41" (UID: "5184916a-2d9b-40cc-9872-9fb0cd055b41"). InnerVolumeSpecName "kube-api-access-xqn5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:21.633537 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.633495    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0720184c-4f19-41d9-871f-b3a208201818-kube-api-access-qqrcl" pod "0720184c-4f19-41d9-871f-b3a208201818" (UID: "0720184c-4f19-41d9-871f-b3a208201818"). InnerVolumeSpecName "kube-api-access-qqrcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:21.639898 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.637580    5639 generic.go:356] "Generic (PLEG): container finished" podID="2aefa8c8-6056-4fd0-8e70-739dbd421750" containerID="35be26f9da8fb5c358c9e2212b04d50dc4a6797d4020c77b6bffcd9ff066fea8" exitCode=60
Nov 02 02:00:21.639898 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.637605    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc" event={"ID":"2aefa8c8-6056-4fd0-8e70-739dbd421750","Type":"ContainerDied","Data":"35be26f9da8fb5c358c9e2212b04d50dc4a6797d4020c77b6bffcd9ff066fea8"}
Nov 02 02:00:21.639898 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.637957    5639 scope.go:122] "RemoveContainer" containerID="35be26f9da8fb5c358c9e2212b04d50dc4a6797d4020c77b6bffcd9ff066fea8"
Nov 02 02:00:21.639898 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:21.638248    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc_odh-ai-gateway-infra(2aefa8c8-6056-4fd0-8e70-739dbd421750)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc" podUID="2aefa8c8-6056-4fd0-8e70-739dbd421750"
Nov 02 02:00:21.658992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.658968    5639 scope.go:122] "RemoveContainer" containerID="5881dfa968046050b057b7f42a8e3c985ef196e08b70cf4feb788f8c91659264"
Nov 02 02:00:21.732528 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.731492    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqn5r\" (UniqueName: \"kubernetes.io/projected/5184916a-2d9b-40cc-9872-9fb0cd055b41-kube-api-access-xqn5r\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:21.732528 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.731526    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0720184c-4f19-41d9-871f-b3a208201818-maas-api-service-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:21.732528 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.731542    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqrcl\" (UniqueName: \"kubernetes.io/projected/0720184c-4f19-41d9-871f-b3a208201818-kube-api-access-qqrcl\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:21.732528 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.731555    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5184916a-2d9b-40cc-9872-9fb0cd055b41-maas-api-service-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:21.736356 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.736334    5639 scope.go:122] "RemoveContainer" containerID="eacb1ebd03cc8564aec55cb6d28edff740e4ebfdf940f0561934ed9f0ffc67aa"
Nov 02 02:00:21.821671 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:21.821637    5639 scope.go:122] "RemoveContainer" containerID="741979e020bee66ec59e9d4c438e8ea0462f013ae6183547a9aca528fda88980"
Nov 02 02:00:21.955717 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:21.955685    5639 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0720184c_4f19_41d9_871f_b3a208201818.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 02:00:22.036288 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.036246    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps"
Nov 02 02:00:22.039992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.039968    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw"
Nov 02 02:00:22.141996 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.140793    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5b031ba-6e9e-4852-abb4-218ab35f22b9-kube-api-access-tvb2w\" (UniqueName: \"kubernetes.io/projected/e5b031ba-6e9e-4852-abb4-218ab35f22b9-kube-api-access-tvb2w\") pod \"e5b031ba-6e9e-4852-abb4-218ab35f22b9\" (UID: \"e5b031ba-6e9e-4852-abb4-218ab35f22b9\") "
Nov 02 02:00:22.141996 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.140861    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5b031ba-6e9e-4852-abb4-218ab35f22b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b031ba-6e9e-4852-abb4-218ab35f22b9-maas-api-service-ca\") pod \"e5b031ba-6e9e-4852-abb4-218ab35f22b9\" (UID: \"e5b031ba-6e9e-4852-abb4-218ab35f22b9\") "
Nov 02 02:00:22.141996 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.140929    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f5659fe-839b-4498-b9eb-2b76d6631417-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f5659fe-839b-4498-b9eb-2b76d6631417-maas-api-service-ca\") pod \"3f5659fe-839b-4498-b9eb-2b76d6631417\" (UID: \"3f5659fe-839b-4498-b9eb-2b76d6631417\") "
Nov 02 02:00:22.141996 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.141050    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f5659fe-839b-4498-b9eb-2b76d6631417-kube-api-access-gqsqq\" (UniqueName: \"kubernetes.io/projected/3f5659fe-839b-4498-b9eb-2b76d6631417-kube-api-access-gqsqq\") pod \"3f5659fe-839b-4498-b9eb-2b76d6631417\" (UID: \"3f5659fe-839b-4498-b9eb-2b76d6631417\") "
Nov 02 02:00:22.144611 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.144581    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b031ba-6e9e-4852-abb4-218ab35f22b9-kube-api-access-tvb2w" pod "e5b031ba-6e9e-4852-abb4-218ab35f22b9" (UID: "e5b031ba-6e9e-4852-abb4-218ab35f22b9"). InnerVolumeSpecName "kube-api-access-tvb2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:22.144933 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.144908    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5b031ba-6e9e-4852-abb4-218ab35f22b9-maas-api-service-ca" pod "e5b031ba-6e9e-4852-abb4-218ab35f22b9" (UID: "e5b031ba-6e9e-4852-abb4-218ab35f22b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:22.145262 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.145240    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f5659fe-839b-4498-b9eb-2b76d6631417-maas-api-service-ca" pod "3f5659fe-839b-4498-b9eb-2b76d6631417" (UID: "3f5659fe-839b-4498-b9eb-2b76d6631417"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:22.147868 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.147844    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5659fe-839b-4498-b9eb-2b76d6631417-kube-api-access-gqsqq" pod "3f5659fe-839b-4498-b9eb-2b76d6631417" (UID: "3f5659fe-839b-4498-b9eb-2b76d6631417"). InnerVolumeSpecName "kube-api-access-gqsqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:22.226513 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.226481    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc"]
Nov 02 02:00:22.230609 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.230585    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88e76d-29893080-5g8rc"]
Nov 02 02:00:22.244544 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.243826    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqsqq\" (UniqueName: \"kubernetes.io/projected/3f5659fe-839b-4498-b9eb-2b76d6631417-kube-api-access-gqsqq\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:22.244544 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.243848    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvb2w\" (UniqueName: \"kubernetes.io/projected/e5b031ba-6e9e-4852-abb4-218ab35f22b9-kube-api-access-tvb2w\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:22.244544 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.243860    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b031ba-6e9e-4852-abb4-218ab35f22b9-maas-api-service-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:22.244544 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.243872    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f5659fe-839b-4498-b9eb-2b76d6631417-maas-api-service-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:22.423448 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.422021    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z"]
Nov 02 02:00:22.427327 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.427240    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c600fc-29893080-d7v2z"]
Nov 02 02:00:22.515318 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.512162    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc"]
Nov 02 02:00:22.650297 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.650246    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw"
Nov 02 02:00:22.650853 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.650297    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw" event={"ID":"3f5659fe-839b-4498-b9eb-2b76d6631417","Type":"ContainerDied","Data":"75e2ed2879b802fde38596ba031e620ae3a653b951c5af78ae1751d761cde0e0"}
Nov 02 02:00:22.650853 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.650374    5639 scope.go:122] "RemoveContainer" containerID="09c43c25ca5d0e1da8fbb8d771535a11b18126f1d19ab8086ab36f57f02fc4db"
Nov 02 02:00:22.657553 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.657525    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps" event={"ID":"e5b031ba-6e9e-4852-abb4-218ab35f22b9","Type":"ContainerDied","Data":"7e7531b8cd3f0985e2328cf4078427861102161e8d7195bc74d969605e5a1bae"}
Nov 02 02:00:22.657689 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.657604    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps"
Nov 02 02:00:22.705441 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.694172    5639 scope.go:122] "RemoveContainer" containerID="db5c76c924c17709888f4d0cd5876bc8757fe116ac29792923b57ce1ebc1b0a7"
Nov 02 02:00:22.849038 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.849016    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt"]
Nov 02 02:00:22.925320 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:22.898712    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-36a1e0-29893080-8msnt"]
Nov 02 02:00:23.149988 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:23.148003    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc"
Nov 02 02:00:23.245646 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:23.245614    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="113179bd-c2bc-4fe6-909e-e1387c459bb9" path="/var/lib/kubelet/pods/113179bd-c2bc-4fe6-909e-e1387c459bb9/volumes"
Nov 02 02:00:23.246091 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:23.246076    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e" path="/var/lib/kubelet/pods/3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e/volumes"
Nov 02 02:00:23.246520 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:23.246505    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fa637e0-ccb8-4713-a16b-f4c353d3cb8f" path="/var/lib/kubelet/pods/3fa637e0-ccb8-4713-a16b-f4c353d3cb8f/volumes"
Nov 02 02:00:23.247206 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:23.247183    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5184916a-2d9b-40cc-9872-9fb0cd055b41" path="/var/lib/kubelet/pods/5184916a-2d9b-40cc-9872-9fb0cd055b41/volumes"
Nov 02 02:00:23.283368 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:23.273763    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aefa8c8-6056-4fd0-8e70-739dbd421750-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aefa8c8-6056-4fd0-8e70-739dbd421750-maas-api-service-ca\") pod \"2aefa8c8-6056-4fd0-8e70-739dbd421750\" (UID: \"2aefa8c8-6056-4fd0-8e70-739dbd421750\") "
Nov 02 02:00:23.283368 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:23.274104    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aefa8c8-6056-4fd0-8e70-739dbd421750-kube-api-access-zdzsz\" (UniqueName: \"kubernetes.io/projected/2aefa8c8-6056-4fd0-8e70-739dbd421750-kube-api-access-zdzsz\") pod \"2aefa8c8-6056-4fd0-8e70-739dbd421750\" (UID: \"2aefa8c8-6056-4fd0-8e70-739dbd421750\") "
Nov 02 02:00:23.308328 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:23.307565    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aefa8c8-6056-4fd0-8e70-739dbd421750-maas-api-service-ca" pod "2aefa8c8-6056-4fd0-8e70-739dbd421750" (UID: "2aefa8c8-6056-4fd0-8e70-739dbd421750"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:23.321902 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:23.321863    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aefa8c8-6056-4fd0-8e70-739dbd421750-kube-api-access-zdzsz" pod "2aefa8c8-6056-4fd0-8e70-739dbd421750" (UID: "2aefa8c8-6056-4fd0-8e70-739dbd421750"). InnerVolumeSpecName "kube-api-access-zdzsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:23.420341 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:23.419659    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdzsz\" (UniqueName: \"kubernetes.io/projected/2aefa8c8-6056-4fd0-8e70-739dbd421750-kube-api-access-zdzsz\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:23.420341 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:23.419695    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aefa8c8-6056-4fd0-8e70-739dbd421750-maas-api-service-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:23.422865 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:23.422834    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh"]
Nov 02 02:00:23.426995 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:23.426970    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a7c06d-29893080-7mnrh"]
Nov 02 02:00:23.719654 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:23.719589    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9"]
Nov 02 02:00:23.724244 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:23.724215    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc" event={"ID":"2aefa8c8-6056-4fd0-8e70-739dbd421750","Type":"ContainerDied","Data":"4090dcba836307fb18c441d0feea248a40a4f3dfd2fd01de5184b409bc75c8f1"}
Nov 02 02:00:23.725114 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:23.724793    5639 scope.go:122] "RemoveContainer" containerID="35be26f9da8fb5c358c9e2212b04d50dc4a6797d4020c77b6bffcd9ff066fea8"
Nov 02 02:00:23.725114 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:23.724389    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc"
Nov 02 02:00:23.725901 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:23.725804    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bb0464-29893080-mdzt9"]
Nov 02 02:00:23.938507 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:23.936033    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2"]
Nov 02 02:00:23.940317 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:23.940257    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893080-npjk2"]
Nov 02 02:00:24.046946 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:24.046830    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps"]
Nov 02 02:00:24.051113 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:24.051086    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d700e9-29893080-9nfps"]
Nov 02 02:00:24.239259 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:24.239227    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc"]
Nov 02 02:00:24.243391 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:24.243363    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-109b07-29893080-bvlrc"]
Nov 02 02:00:24.415960 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:24.415924    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw"]
Nov 02 02:00:24.422252 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:24.422221    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-07ac70d0-29893080-ffkcw"]
Nov 02 02:00:24.501776 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:24.492970    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd"]
Nov 02 02:00:24.616378 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:24.614041    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm"]
Nov 02 02:00:25.252648 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:25.252617    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0720184c-4f19-41d9-871f-b3a208201818" path="/var/lib/kubelet/pods/0720184c-4f19-41d9-871f-b3a208201818/volumes"
Nov 02 02:00:25.253561 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:25.253541    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aefa8c8-6056-4fd0-8e70-739dbd421750" path="/var/lib/kubelet/pods/2aefa8c8-6056-4fd0-8e70-739dbd421750/volumes"
Nov 02 02:00:25.254053 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:25.254040    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c0d592c-c444-4131-ac9c-13e3980e5a6d" path="/var/lib/kubelet/pods/2c0d592c-c444-4131-ac9c-13e3980e5a6d/volumes"
Nov 02 02:00:25.254856 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:25.254842    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f5659fe-839b-4498-b9eb-2b76d6631417" path="/var/lib/kubelet/pods/3f5659fe-839b-4498-b9eb-2b76d6631417/volumes"
Nov 02 02:00:25.255357 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:25.255343    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d96d25d2-704b-40d5-9a6b-116c4eb3a4a4" path="/var/lib/kubelet/pods/d96d25d2-704b-40d5-9a6b-116c4eb3a4a4/volumes"
Nov 02 02:00:25.255827 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:25.255813    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5b031ba-6e9e-4852-abb4-218ab35f22b9" path="/var/lib/kubelet/pods/e5b031ba-6e9e-4852-abb4-218ab35f22b9/volumes"
Nov 02 02:00:27.425510 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:27.425470    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:00:29.494114 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:29.494074    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd" podUID="ff588a60-de17-4af0-9710-c8ec8fad7ef6" containerName="payload-processing" containerID="cri-o://d90987704b4c7cfd628a0f0d2fe34c0e4a651cf6427f1bdaf88bf20455b0e441" gracePeriod=25
Nov 02 02:00:29.615106 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:29.615016    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm" podUID="b0b1ff83-fc50-4730-a4ab-e566d6c04d30" containerName="payload-pre-processing" containerID="cri-o://716aea416f18197d1c2451aa4b4b26687112ba9e73d214d24245c82be80055d2" gracePeriod=25
Nov 02 02:00:29.842544 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:29.842510    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd"
Nov 02 02:00:29.944170 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:29.944148    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm"
Nov 02 02:00:29.952828 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:29.952802    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff588a60-de17-4af0-9710-c8ec8fad7ef6-kube-api-access-4mbfn\" (UniqueName: \"kubernetes.io/projected/ff588a60-de17-4af0-9710-c8ec8fad7ef6-kube-api-access-4mbfn\") pod \"ff588a60-de17-4af0-9710-c8ec8fad7ef6\" (UID: \"ff588a60-de17-4af0-9710-c8ec8fad7ef6\") "
Nov 02 02:00:29.952996 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:29.952982    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff588a60-de17-4af0-9710-c8ec8fad7ef6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff588a60-de17-4af0-9710-c8ec8fad7ef6-plugins-config-volume\") pod \"ff588a60-de17-4af0-9710-c8ec8fad7ef6\" (UID: \"ff588a60-de17-4af0-9710-c8ec8fad7ef6\") "
Nov 02 02:00:29.953388 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:29.953366    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff588a60-de17-4af0-9710-c8ec8fad7ef6-plugins-config-volume" pod "ff588a60-de17-4af0-9710-c8ec8fad7ef6" (UID: "ff588a60-de17-4af0-9710-c8ec8fad7ef6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:29.953779 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:29.953764    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff588a60-de17-4af0-9710-c8ec8fad7ef6-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:29.955053 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:29.955031    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff588a60-de17-4af0-9710-c8ec8fad7ef6-kube-api-access-4mbfn" pod "ff588a60-de17-4af0-9710-c8ec8fad7ef6" (UID: "ff588a60-de17-4af0-9710-c8ec8fad7ef6"). InnerVolumeSpecName "kube-api-access-4mbfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:29.981956 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:29.981901    5639 generic.go:356] "Generic (PLEG): container finished" podID="b0b1ff83-fc50-4730-a4ab-e566d6c04d30" containerID="716aea416f18197d1c2451aa4b4b26687112ba9e73d214d24245c82be80055d2" exitCode=0
Nov 02 02:00:29.982410 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:29.982391    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm"
Nov 02 02:00:29.982507 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:29.982453    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm" event={"ID":"b0b1ff83-fc50-4730-a4ab-e566d6c04d30","Type":"ContainerDied","Data":"716aea416f18197d1c2451aa4b4b26687112ba9e73d214d24245c82be80055d2"}
Nov 02 02:00:29.982567 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:29.982537    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm" event={"ID":"b0b1ff83-fc50-4730-a4ab-e566d6c04d30","Type":"ContainerDied","Data":"5cfbcf24be756891485ce72c5cc248fd6df25645d639adc7c37b8a91aa6d6497"}
Nov 02 02:00:29.982630 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:29.982571    5639 scope.go:122] "RemoveContainer" containerID="716aea416f18197d1c2451aa4b4b26687112ba9e73d214d24245c82be80055d2"
Nov 02 02:00:29.984403 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:29.984295    5639 generic.go:356] "Generic (PLEG): container finished" podID="ff588a60-de17-4af0-9710-c8ec8fad7ef6" containerID="d90987704b4c7cfd628a0f0d2fe34c0e4a651cf6427f1bdaf88bf20455b0e441" exitCode=0
Nov 02 02:00:29.984403 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:29.984341    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd" event={"ID":"ff588a60-de17-4af0-9710-c8ec8fad7ef6","Type":"ContainerDied","Data":"d90987704b4c7cfd628a0f0d2fe34c0e4a651cf6427f1bdaf88bf20455b0e441"}
Nov 02 02:00:29.984403 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:29.984382    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd" event={"ID":"ff588a60-de17-4af0-9710-c8ec8fad7ef6","Type":"ContainerDied","Data":"65a18f7b55a26a8110bf3bf45096521a02cd467828a346b6ba4dd8d14e11bf27"}
Nov 02 02:00:29.984403 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:29.984417    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd"
Nov 02 02:00:30.002592 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:30.002568    5639 scope.go:122] "RemoveContainer" containerID="716aea416f18197d1c2451aa4b4b26687112ba9e73d214d24245c82be80055d2"
Nov 02 02:00:30.002963 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:30.002938    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"716aea416f18197d1c2451aa4b4b26687112ba9e73d214d24245c82be80055d2\": container with ID starting with 716aea416f18197d1c2451aa4b4b26687112ba9e73d214d24245c82be80055d2 not found: ID does not exist" containerID="716aea416f18197d1c2451aa4b4b26687112ba9e73d214d24245c82be80055d2"
Nov 02 02:00:30.003065 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:30.002976    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716aea416f18197d1c2451aa4b4b26687112ba9e73d214d24245c82be80055d2"} err="failed to get container status \"716aea416f18197d1c2451aa4b4b26687112ba9e73d214d24245c82be80055d2\": rpc error: code = NotFound desc = could not find container \"716aea416f18197d1c2451aa4b4b26687112ba9e73d214d24245c82be80055d2\": container with ID starting with 716aea416f18197d1c2451aa4b4b26687112ba9e73d214d24245c82be80055d2 not found: ID does not exist"
Nov 02 02:00:30.003065 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:30.003000    5639 scope.go:122] "RemoveContainer" containerID="d90987704b4c7cfd628a0f0d2fe34c0e4a651cf6427f1bdaf88bf20455b0e441"
Nov 02 02:00:30.058700 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:30.057168    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b1ff83-fc50-4730-a4ab-e566d6c04d30-kube-api-access-wpkj7\" (UniqueName: \"kubernetes.io/projected/b0b1ff83-fc50-4730-a4ab-e566d6c04d30-kube-api-access-wpkj7\") pod \"b0b1ff83-fc50-4730-a4ab-e566d6c04d30\" (UID: \"b0b1ff83-fc50-4730-a4ab-e566d6c04d30\") "
Nov 02 02:00:30.058700 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:30.057376    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0b1ff83-fc50-4730-a4ab-e566d6c04d30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0b1ff83-fc50-4730-a4ab-e566d6c04d30-plugins-config-volume\") pod \"b0b1ff83-fc50-4730-a4ab-e566d6c04d30\" (UID: \"b0b1ff83-fc50-4730-a4ab-e566d6c04d30\") "
Nov 02 02:00:30.058700 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:30.058023    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mbfn\" (UniqueName: \"kubernetes.io/projected/ff588a60-de17-4af0-9710-c8ec8fad7ef6-kube-api-access-4mbfn\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:30.074108 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:30.074083    5639 scope.go:122] "RemoveContainer" containerID="d90987704b4c7cfd628a0f0d2fe34c0e4a651cf6427f1bdaf88bf20455b0e441"
Nov 02 02:00:30.076474 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:30.076401    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d90987704b4c7cfd628a0f0d2fe34c0e4a651cf6427f1bdaf88bf20455b0e441\": container with ID starting with d90987704b4c7cfd628a0f0d2fe34c0e4a651cf6427f1bdaf88bf20455b0e441 not found: ID does not exist" containerID="d90987704b4c7cfd628a0f0d2fe34c0e4a651cf6427f1bdaf88bf20455b0e441"
Nov 02 02:00:30.076677 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:30.076636    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d90987704b4c7cfd628a0f0d2fe34c0e4a651cf6427f1bdaf88bf20455b0e441"} err="failed to get container status \"d90987704b4c7cfd628a0f0d2fe34c0e4a651cf6427f1bdaf88bf20455b0e441\": rpc error: code = NotFound desc = could not find container \"d90987704b4c7cfd628a0f0d2fe34c0e4a651cf6427f1bdaf88bf20455b0e441\": container with ID starting with d90987704b4c7cfd628a0f0d2fe34c0e4a651cf6427f1bdaf88bf20455b0e441 not found: ID does not exist"
Nov 02 02:00:30.084712 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:30.084673    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0b1ff83-fc50-4730-a4ab-e566d6c04d30-plugins-config-volume" pod "b0b1ff83-fc50-4730-a4ab-e566d6c04d30" (UID: "b0b1ff83-fc50-4730-a4ab-e566d6c04d30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:30.095192 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:30.095157    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b1ff83-fc50-4730-a4ab-e566d6c04d30-kube-api-access-wpkj7" pod "b0b1ff83-fc50-4730-a4ab-e566d6c04d30" (UID: "b0b1ff83-fc50-4730-a4ab-e566d6c04d30"). InnerVolumeSpecName "kube-api-access-wpkj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:30.238025 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:30.159160    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpkj7\" (UniqueName: \"kubernetes.io/projected/b0b1ff83-fc50-4730-a4ab-e566d6c04d30-kube-api-access-wpkj7\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:30.238025 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:30.159183    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0b1ff83-fc50-4730-a4ab-e566d6c04d30-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:30.423200 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:30.262653    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd"]
Nov 02 02:00:30.423200 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:30.267382    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-109b07-69bb9ddc6f-g26jd"]
Nov 02 02:00:30.423200 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:30.406232    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm"]
Nov 02 02:00:30.423200 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:30.410724    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-109b07-799fc87f5f-clvgm"]
Nov 02 02:00:30.993417 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:30.993383    5639 generic.go:356] "Generic (PLEG): container finished" podID="25cd82fe-a02f-45cf-9a41-bdbaea642b30" containerID="e4a8674d8a142bb9b0d93ad7b91bb3bf25b3066656b2892728e2b62aa4a9ca6f" exitCode=0
Nov 02 02:00:30.993901 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:30.993458    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw" event={"ID":"25cd82fe-a02f-45cf-9a41-bdbaea642b30","Type":"ContainerDied","Data":"e4a8674d8a142bb9b0d93ad7b91bb3bf25b3066656b2892728e2b62aa4a9ca6f"}
Nov 02 02:00:30.993983 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:30.993965    5639 scope.go:122] "RemoveContainer" containerID="e4a8674d8a142bb9b0d93ad7b91bb3bf25b3066656b2892728e2b62aa4a9ca6f"
Nov 02 02:00:30.996081 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:30.996046    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:00:30.996188 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:30.996143    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:00:30.996188 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:30.996185    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:00:30.997797 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:30.997776    5639 generic.go:356] "Generic (PLEG): container finished" podID="85c9d8c5-bc73-4b32-b543-adc3d9509702" containerID="7307ad5782c6bf3c83a8d5bdf5d74dceb401a2a1c965c4c0615949c242c1e3f6" exitCode=0
Nov 02 02:00:30.997924 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:30.997903    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276" event={"ID":"85c9d8c5-bc73-4b32-b543-adc3d9509702","Type":"ContainerDied","Data":"7307ad5782c6bf3c83a8d5bdf5d74dceb401a2a1c965c4c0615949c242c1e3f6"}
Nov 02 02:00:30.998344 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:30.998326    5639 scope.go:122] "RemoveContainer" containerID="7307ad5782c6bf3c83a8d5bdf5d74dceb401a2a1c965c4c0615949c242c1e3f6"
Nov 02 02:00:31.000645 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:31.000616    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:00:31.000742 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:31.000717    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:00:31.000975 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:31.000768    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:00:31.242302 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:31.241603    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0b1ff83-fc50-4730-a4ab-e566d6c04d30" path="/var/lib/kubelet/pods/b0b1ff83-fc50-4730-a4ab-e566d6c04d30/volumes"
Nov 02 02:00:31.242859 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:31.242832    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff588a60-de17-4af0-9710-c8ec8fad7ef6" path="/var/lib/kubelet/pods/ff588a60-de17-4af0-9710-c8ec8fad7ef6/volumes"
Nov 02 02:00:32.020883 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:32.020854    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276" event={"ID":"85c9d8c5-bc73-4b32-b543-adc3d9509702","Type":"ContainerStarted","Data":"95b450db3daa5325868975e56fc1e70ef6b3d102c887199edd945ee6875fe348"}
Nov 02 02:00:32.023171 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:32.023144    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw" event={"ID":"25cd82fe-a02f-45cf-9a41-bdbaea642b30","Type":"ContainerStarted","Data":"e12caf8cd904f257f4df4da84b5ed8b236f6e573f0bbccc85f37697b9b94c095"}
Nov 02 02:00:32.368304 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:32.365110    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 02:00:32.368304 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:32.365249    5639 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 02 02:00:32.368304 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:32.365342    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276" podUID="85c9d8c5-bc73-4b32-b543-adc3d9509702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 02 02:00:32.393304 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:32.392380    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 02:00:32.430236 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:32.430204    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 02:00:32.431111 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:32.431090    5639 patch_prober.go:29] interesting pod/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 02 02:00:32.431172 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:32.431132    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw" podUID="25cd82fe-a02f-45cf-9a41-bdbaea642b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 02 02:00:33.380851 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:33.380809    5639 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 02 02:00:33.381538 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:33.381502    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276" podUID="85c9d8c5-bc73-4b32-b543-adc3d9509702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 02 02:00:33.431029 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:33.430989    5639 patch_prober.go:29] interesting pod/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 02 02:00:33.431246 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:33.431045    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw" podUID="25cd82fe-a02f-45cf-9a41-bdbaea642b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 02 02:00:34.367309 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:34.363685    5639 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 02 02:00:34.367309 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:34.363748    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276" podUID="85c9d8c5-bc73-4b32-b543-adc3d9509702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 02 02:00:34.430245 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:34.430207    5639 patch_prober.go:29] interesting pod/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 02 02:00:34.430691 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:34.430334    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw" podUID="25cd82fe-a02f-45cf-9a41-bdbaea642b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 02 02:00:34.619601 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:34.619510    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 02:00:35.388310 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:35.385819    5639 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 02 02:00:35.388310 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:35.385883    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276" podUID="85c9d8c5-bc73-4b32-b543-adc3d9509702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 02 02:00:35.431044 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:35.431013    5639 patch_prober.go:29] interesting pod/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 02 02:00:35.431473 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:35.431074    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw" podUID="25cd82fe-a02f-45cf-9a41-bdbaea642b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 02 02:00:36.367311 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:36.363633    5639 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 02 02:00:36.367311 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:36.363729    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276" podUID="85c9d8c5-bc73-4b32-b543-adc3d9509702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 02 02:00:36.430478 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:36.430435    5639 patch_prober.go:29] interesting pod/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 02 02:00:36.430679 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:36.430495    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw" podUID="25cd82fe-a02f-45cf-9a41-bdbaea642b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 02 02:00:37.368691 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:37.368663    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 02:00:37.369791 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:37.369771    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 02:00:37.435731 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:37.435697    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 02:00:37.436687 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:37.436661    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 02:00:37.870820 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:37.870792    5639 scope.go:122] "RemoveContainer" containerID="71da0847bc55babba7a758308b5d1f2348c91ef7ae2bd959f3e5ea460f74db60"
Nov 02 02:00:37.881719 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:37.881699    5639 scope.go:122] "RemoveContainer" containerID="9c51f0544405a16a33b268072e38d2f87d75ce7b90c3962ad208241edeca117c"
Nov 02 02:00:40.051732 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:40.051700    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fce56afe-g94dz"]
Nov 02 02:00:40.101250 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:40.101219    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fce56afe-g94dz"]
Nov 02 02:00:41.221517 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:00:41.221487    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3066172f-0a14-486e-9cba-a2177d0760ce" path="/var/lib/kubelet/pods/3066172f-0a14-486e-9cba-a2177d0760ce/volumes"
Nov 02 02:00:57.421844 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:00:57.421793    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:01:03.455831 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.455791    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd"]
Nov 02 02:01:03.456917 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.456895    5639 cpu_manager.go:414] "containerMap: removing container" podUID="0720184c-4f19-41d9-871f-b3a208201818" containerName="cleanup"
Nov 02 02:01:03.456992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.456921    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0720184c-4f19-41d9-871f-b3a208201818" containerName="cleanup"
Nov 02 02:01:03.456992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.456938    5639 cpu_manager.go:414] "containerMap: removing container" podUID="2aefa8c8-6056-4fd0-8e70-739dbd421750" containerName="cleanup"
Nov 02 02:01:03.456992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.456945    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aefa8c8-6056-4fd0-8e70-739dbd421750" containerName="cleanup"
Nov 02 02:01:03.456992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.456963    5639 cpu_manager.go:414] "containerMap: removing container" podUID="0720184c-4f19-41d9-871f-b3a208201818" containerName="cleanup"
Nov 02 02:01:03.456992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.456970    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0720184c-4f19-41d9-871f-b3a208201818" containerName="cleanup"
Nov 02 02:01:03.456992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.456985    5639 cpu_manager.go:414] "containerMap: removing container" podUID="d96d25d2-704b-40d5-9a6b-116c4eb3a4a4" containerName="cleanup"
Nov 02 02:01:03.456992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.456992    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d96d25d2-704b-40d5-9a6b-116c4eb3a4a4" containerName="cleanup"
Nov 02 02:01:03.457218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457001    5639 cpu_manager.go:414] "containerMap: removing container" podUID="ff588a60-de17-4af0-9710-c8ec8fad7ef6" containerName="payload-processing"
Nov 02 02:01:03.457218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457008    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff588a60-de17-4af0-9710-c8ec8fad7ef6" containerName="payload-processing"
Nov 02 02:01:03.457218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457023    5639 cpu_manager.go:414] "containerMap: removing container" podUID="5dabb512-1926-413b-87ab-2a9c87e685f7" containerName="cleanup"
Nov 02 02:01:03.457218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457030    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dabb512-1926-413b-87ab-2a9c87e685f7" containerName="cleanup"
Nov 02 02:01:03.457218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457045    5639 cpu_manager.go:414] "containerMap: removing container" podUID="2c0d592c-c444-4131-ac9c-13e3980e5a6d" containerName="cleanup"
Nov 02 02:01:03.457218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457052    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c0d592c-c444-4131-ac9c-13e3980e5a6d" containerName="cleanup"
Nov 02 02:01:03.457218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457060    5639 cpu_manager.go:414] "containerMap: removing container" podUID="113179bd-c2bc-4fe6-909e-e1387c459bb9" containerName="cleanup"
Nov 02 02:01:03.457218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457067    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="113179bd-c2bc-4fe6-909e-e1387c459bb9" containerName="cleanup"
Nov 02 02:01:03.457218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457082    5639 cpu_manager.go:414] "containerMap: removing container" podUID="3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e" containerName="cleanup"
Nov 02 02:01:03.457218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457088    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e" containerName="cleanup"
Nov 02 02:01:03.457218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457108    5639 cpu_manager.go:414] "containerMap: removing container" podUID="2aefa8c8-6056-4fd0-8e70-739dbd421750" containerName="cleanup"
Nov 02 02:01:03.457218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457114    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aefa8c8-6056-4fd0-8e70-739dbd421750" containerName="cleanup"
Nov 02 02:01:03.457218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457123    5639 cpu_manager.go:414] "containerMap: removing container" podUID="113179bd-c2bc-4fe6-909e-e1387c459bb9" containerName="cleanup"
Nov 02 02:01:03.457218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457129    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="113179bd-c2bc-4fe6-909e-e1387c459bb9" containerName="cleanup"
Nov 02 02:01:03.457218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457142    5639 cpu_manager.go:414] "containerMap: removing container" podUID="3fa637e0-ccb8-4713-a16b-f4c353d3cb8f" containerName="cleanup"
Nov 02 02:01:03.457218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457149    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa637e0-ccb8-4713-a16b-f4c353d3cb8f" containerName="cleanup"
Nov 02 02:01:03.457218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457160    5639 cpu_manager.go:414] "containerMap: removing container" podUID="c471528a-da4f-46c6-b48d-3404eec749c0" containerName="revoke-keys"
Nov 02 02:01:03.457218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457167    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c471528a-da4f-46c6-b48d-3404eec749c0" containerName="revoke-keys"
Nov 02 02:01:03.457218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457181    5639 cpu_manager.go:414] "containerMap: removing container" podUID="e5b031ba-6e9e-4852-abb4-218ab35f22b9" containerName="cleanup"
Nov 02 02:01:03.457218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457188    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5b031ba-6e9e-4852-abb4-218ab35f22b9" containerName="cleanup"
Nov 02 02:01:03.457218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457201    5639 cpu_manager.go:414] "containerMap: removing container" podUID="b0b1ff83-fc50-4730-a4ab-e566d6c04d30" containerName="payload-pre-processing"
Nov 02 02:01:03.457218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457209    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0b1ff83-fc50-4730-a4ab-e566d6c04d30" containerName="payload-pre-processing"
Nov 02 02:01:03.457218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457221    5639 cpu_manager.go:414] "containerMap: removing container" podUID="3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457228    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457241    5639 cpu_manager.go:414] "containerMap: removing container" podUID="2c0d592c-c444-4131-ac9c-13e3980e5a6d" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457250    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c0d592c-c444-4131-ac9c-13e3980e5a6d" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457259    5639 cpu_manager.go:414] "containerMap: removing container" podUID="3fa637e0-ccb8-4713-a16b-f4c353d3cb8f" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457267    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa637e0-ccb8-4713-a16b-f4c353d3cb8f" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457295    5639 cpu_manager.go:414] "containerMap: removing container" podUID="3f5659fe-839b-4498-b9eb-2b76d6631417" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457302    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f5659fe-839b-4498-b9eb-2b76d6631417" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457317    5639 cpu_manager.go:414] "containerMap: removing container" podUID="5dabb512-1926-413b-87ab-2a9c87e685f7" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457324    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dabb512-1926-413b-87ab-2a9c87e685f7" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457337    5639 cpu_manager.go:414] "containerMap: removing container" podUID="5dabb512-1926-413b-87ab-2a9c87e685f7" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457344    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dabb512-1926-413b-87ab-2a9c87e685f7" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457372    5639 cpu_manager.go:414] "containerMap: removing container" podUID="e5b031ba-6e9e-4852-abb4-218ab35f22b9" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457379    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5b031ba-6e9e-4852-abb4-218ab35f22b9" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457390    5639 cpu_manager.go:414] "containerMap: removing container" podUID="b669073a-3244-4c17-9255-197abd78c544" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457396    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b669073a-3244-4c17-9255-197abd78c544" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457411    5639 cpu_manager.go:414] "containerMap: removing container" podUID="e5b031ba-6e9e-4852-abb4-218ab35f22b9" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457418    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5b031ba-6e9e-4852-abb4-218ab35f22b9" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457431    5639 cpu_manager.go:414] "containerMap: removing container" podUID="5184916a-2d9b-40cc-9872-9fb0cd055b41" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457438    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5184916a-2d9b-40cc-9872-9fb0cd055b41" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457452    5639 cpu_manager.go:414] "containerMap: removing container" podUID="5184916a-2d9b-40cc-9872-9fb0cd055b41" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457458    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5184916a-2d9b-40cc-9872-9fb0cd055b41" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457469    5639 cpu_manager.go:414] "containerMap: removing container" podUID="d96d25d2-704b-40d5-9a6b-116c4eb3a4a4" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457475    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d96d25d2-704b-40d5-9a6b-116c4eb3a4a4" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457490    5639 cpu_manager.go:414] "containerMap: removing container" podUID="3f5659fe-839b-4498-b9eb-2b76d6631417" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457496    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f5659fe-839b-4498-b9eb-2b76d6631417" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457505    5639 cpu_manager.go:414] "containerMap: removing container" podUID="b669073a-3244-4c17-9255-197abd78c544" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457511    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b669073a-3244-4c17-9255-197abd78c544" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457523    5639 cpu_manager.go:414] "containerMap: removing container" podUID="b669073a-3244-4c17-9255-197abd78c544" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457530    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b669073a-3244-4c17-9255-197abd78c544" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457542    5639 cpu_manager.go:414] "containerMap: removing container" podUID="5184916a-2d9b-40cc-9872-9fb0cd055b41" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457548    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5184916a-2d9b-40cc-9872-9fb0cd055b41" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457709    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dabb512-1926-413b-87ab-2a9c87e685f7" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457725    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dabb512-1926-413b-87ab-2a9c87e685f7" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457739    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c0d592c-c444-4131-ac9c-13e3980e5a6d" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457751    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5b031ba-6e9e-4852-abb4-218ab35f22b9" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457760    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="b669073a-3244-4c17-9255-197abd78c544" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457770    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="d96d25d2-704b-40d5-9a6b-116c4eb3a4a4" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457780    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f5659fe-839b-4498-b9eb-2b76d6631417" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457787    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5b031ba-6e9e-4852-abb4-218ab35f22b9" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457798    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="5184916a-2d9b-40cc-9872-9fb0cd055b41" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457805    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="5184916a-2d9b-40cc-9872-9fb0cd055b41" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457815    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="d96d25d2-704b-40d5-9a6b-116c4eb3a4a4" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457825    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa637e0-ccb8-4713-a16b-f4c353d3cb8f" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457833    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f5659fe-839b-4498-b9eb-2b76d6631417" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457840    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="b669073a-3244-4c17-9255-197abd78c544" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457850    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="b669073a-3244-4c17-9255-197abd78c544" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457861    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="113179bd-c2bc-4fe6-909e-e1387c459bb9" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457870    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="0720184c-4f19-41d9-871f-b3a208201818" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457877    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aefa8c8-6056-4fd0-8e70-739dbd421750" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457886    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="0720184c-4f19-41d9-871f-b3a208201818" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457894    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="d96d25d2-704b-40d5-9a6b-116c4eb3a4a4" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457905    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff588a60-de17-4af0-9710-c8ec8fad7ef6" containerName="payload-processing"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457917    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="113179bd-c2bc-4fe6-909e-e1387c459bb9" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457926    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457935    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aefa8c8-6056-4fd0-8e70-739dbd421750" containerName="cleanup"
Nov 02 02:01:03.457927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457943    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="113179bd-c2bc-4fe6-909e-e1387c459bb9" containerName="cleanup"
Nov 02 02:01:03.459731 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457954    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa637e0-ccb8-4713-a16b-f4c353d3cb8f" containerName="cleanup"
Nov 02 02:01:03.459731 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457964    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="c471528a-da4f-46c6-b48d-3404eec749c0" containerName="revoke-keys"
Nov 02 02:01:03.459731 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457974    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e" containerName="cleanup"
Nov 02 02:01:03.459731 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457985    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0b1ff83-fc50-4730-a4ab-e566d6c04d30" containerName="payload-pre-processing"
Nov 02 02:01:03.459731 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.457994    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e" containerName="cleanup"
Nov 02 02:01:03.459731 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.458002    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c0d592c-c444-4131-ac9c-13e3980e5a6d" containerName="cleanup"
Nov 02 02:01:03.459731 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.458011    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f5659fe-839b-4498-b9eb-2b76d6631417" containerName="cleanup"
Nov 02 02:01:03.461397 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.461380    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd"
Nov 02 02:01:03.517006 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.516972    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd"]
Nov 02 02:01:03.561508 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.561465    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:01:03.561508 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.561504    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:01:03.561781 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.561508    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:01:03.561781 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.561530    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:01:03.565489 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.565463    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7s9z\" (UniqueName: \"kubernetes.io/projected/bccefd52-b090-41c8-88eb-83d79277cd74-kube-api-access-p7s9z\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd\" (UID: \"bccefd52-b090-41c8-88eb-83d79277cd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd"
Nov 02 02:01:03.565683 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.565664    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bccefd52-b090-41c8-88eb-83d79277cd74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd\" (UID: \"bccefd52-b090-41c8-88eb-83d79277cd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd"
Nov 02 02:01:03.565740 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.565691    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bccefd52-b090-41c8-88eb-83d79277cd74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd\" (UID: \"bccefd52-b090-41c8-88eb-83d79277cd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd"
Nov 02 02:01:03.667210 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.667169    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bccefd52-b090-41c8-88eb-83d79277cd74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd\" (UID: \"bccefd52-b090-41c8-88eb-83d79277cd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd"
Nov 02 02:01:03.667210 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.667207    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bccefd52-b090-41c8-88eb-83d79277cd74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd\" (UID: \"bccefd52-b090-41c8-88eb-83d79277cd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd"
Nov 02 02:01:03.667482 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.667398    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7s9z\" (UniqueName: \"kubernetes.io/projected/bccefd52-b090-41c8-88eb-83d79277cd74-kube-api-access-p7s9z\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd\" (UID: \"bccefd52-b090-41c8-88eb-83d79277cd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd"
Nov 02 02:01:03.667863 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.667832    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bccefd52-b090-41c8-88eb-83d79277cd74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd\" (UID: \"bccefd52-b090-41c8-88eb-83d79277cd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd"
Nov 02 02:01:03.669873 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.669842    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bccefd52-b090-41c8-88eb-83d79277cd74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd\" (UID: \"bccefd52-b090-41c8-88eb-83d79277cd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd"
Nov 02 02:01:03.728372 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.728298    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7s9z\" (UniqueName: \"kubernetes.io/projected/bccefd52-b090-41c8-88eb-83d79277cd74-kube-api-access-p7s9z\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd\" (UID: \"bccefd52-b090-41c8-88eb-83d79277cd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd"
Nov 02 02:01:03.774812 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:03.774774    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd"
Nov 02 02:01:04.161113 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:04.160937    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd"]
Nov 02 02:01:04.347974 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:04.347942    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd" event={"ID":"bccefd52-b090-41c8-88eb-83d79277cd74","Type":"ContainerStarted","Data":"6bb232965aa7e4d72b289bc2864dec0d4f23e034a208a7af383067292b82fa58"}
Nov 02 02:01:05.357710 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:05.357677    5639 generic.go:356] "Generic (PLEG): container finished" podID="bccefd52-b090-41c8-88eb-83d79277cd74" containerID="897f3fa51e2f6a6ae30aaad99dca841ffb8917b8ccede2b668f5288709661e3d" exitCode=0
Nov 02 02:01:05.358142 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:05.357723    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd" event={"ID":"bccefd52-b090-41c8-88eb-83d79277cd74","Type":"ContainerDied","Data":"897f3fa51e2f6a6ae30aaad99dca841ffb8917b8ccede2b668f5288709661e3d"}
Nov 02 02:01:06.536735 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:06.536708    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd"
Nov 02 02:01:06.611645 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:06.611611    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bccefd52-b090-41c8-88eb-83d79277cd74-kube-api-access-p7s9z\" (UniqueName: \"kubernetes.io/projected/bccefd52-b090-41c8-88eb-83d79277cd74-kube-api-access-p7s9z\") pod \"bccefd52-b090-41c8-88eb-83d79277cd74\" (UID: \"bccefd52-b090-41c8-88eb-83d79277cd74\") "
Nov 02 02:01:06.611801 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:06.611777    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bccefd52-b090-41c8-88eb-83d79277cd74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bccefd52-b090-41c8-88eb-83d79277cd74-maas-api-xks-ca\") pod \"bccefd52-b090-41c8-88eb-83d79277cd74\" (UID: \"bccefd52-b090-41c8-88eb-83d79277cd74\") "
Nov 02 02:01:06.611879 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:06.611811    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bccefd52-b090-41c8-88eb-83d79277cd74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bccefd52-b090-41c8-88eb-83d79277cd74-maas-api-openshift-ca\") pod \"bccefd52-b090-41c8-88eb-83d79277cd74\" (UID: \"bccefd52-b090-41c8-88eb-83d79277cd74\") "
Nov 02 02:01:06.612219 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:06.612193    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bccefd52-b090-41c8-88eb-83d79277cd74-maas-api-openshift-ca" pod "bccefd52-b090-41c8-88eb-83d79277cd74" (UID: "bccefd52-b090-41c8-88eb-83d79277cd74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:06.612345 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:06.612325    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bccefd52-b090-41c8-88eb-83d79277cd74-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:06.613939 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:06.613913    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bccefd52-b090-41c8-88eb-83d79277cd74-kube-api-access-p7s9z" pod "bccefd52-b090-41c8-88eb-83d79277cd74" (UID: "bccefd52-b090-41c8-88eb-83d79277cd74"). InnerVolumeSpecName "kube-api-access-p7s9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:06.614028 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:06.614008    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bccefd52-b090-41c8-88eb-83d79277cd74-maas-api-xks-ca" pod "bccefd52-b090-41c8-88eb-83d79277cd74" (UID: "bccefd52-b090-41c8-88eb-83d79277cd74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:01:06.713777 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:06.713742    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7s9z\" (UniqueName: \"kubernetes.io/projected/bccefd52-b090-41c8-88eb-83d79277cd74-kube-api-access-p7s9z\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:06.713777 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:06.713771    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bccefd52-b090-41c8-88eb-83d79277cd74-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:07.373837 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:07.373803    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd"
Nov 02 02:01:07.373837 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:07.373830    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd" event={"ID":"bccefd52-b090-41c8-88eb-83d79277cd74","Type":"ContainerDied","Data":"6bb232965aa7e4d72b289bc2864dec0d4f23e034a208a7af383067292b82fa58"}
Nov 02 02:01:07.374072 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:07.373879    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bb232965aa7e4d72b289bc2864dec0d4f23e034a208a7af383067292b82fa58"
Nov 02 02:01:10.027840 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:10.027789    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:01:13.139209 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:13.139172    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96"]
Nov 02 02:01:13.188381 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:13.188331    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a7c06d-r8t96"]
Nov 02 02:01:13.221488 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:13.221446    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbcceaa5-440d-4605-b0ed-7b0836ceb2bf" path="/var/lib/kubelet/pods/fbcceaa5-440d-4605-b0ed-7b0836ceb2bf/volumes"
Nov 02 02:01:14.950690 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:14.950653    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw"]
Nov 02 02:01:15.058832 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:15.058794    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs"]
Nov 02 02:01:19.951324 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:19.951249    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw" podUID="43fdb542-4aa2-47d6-9d83-f29f584cf771" containerName="payload-processing" containerID="cri-o://68ed8a26ab82b43d6a6ab2a3ee684befae8c8a35deb25c862a084f135ab968af" gracePeriod=25
Nov 02 02:01:20.059328 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.059254    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs" podUID="8815f4d6-021e-4c5f-9638-5c53d8470c5d" containerName="payload-pre-processing" containerID="cri-o://3049b3a58f82c4549afdc6cc77d5b89cebd558dedb9f5d9f34dd6c19116ded74" gracePeriod=25
Nov 02 02:01:20.283348 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.283210    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw"
Nov 02 02:01:20.355224 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.355185    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43fdb542-4aa2-47d6-9d83-f29f584cf771-kube-api-access-67q72\" (UniqueName: \"kubernetes.io/projected/43fdb542-4aa2-47d6-9d83-f29f584cf771-kube-api-access-67q72\") pod \"43fdb542-4aa2-47d6-9d83-f29f584cf771\" (UID: \"43fdb542-4aa2-47d6-9d83-f29f584cf771\") "
Nov 02 02:01:20.355598 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.355580    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43fdb542-4aa2-47d6-9d83-f29f584cf771-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43fdb542-4aa2-47d6-9d83-f29f584cf771-plugins-config-volume\") pod \"43fdb542-4aa2-47d6-9d83-f29f584cf771\" (UID: \"43fdb542-4aa2-47d6-9d83-f29f584cf771\") "
Nov 02 02:01:20.355905 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.355881    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43fdb542-4aa2-47d6-9d83-f29f584cf771-plugins-config-volume" pod "43fdb542-4aa2-47d6-9d83-f29f584cf771" (UID: "43fdb542-4aa2-47d6-9d83-f29f584cf771"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:20.356262 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.356247    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43fdb542-4aa2-47d6-9d83-f29f584cf771-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:20.357649 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.357615    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43fdb542-4aa2-47d6-9d83-f29f584cf771-kube-api-access-67q72" pod "43fdb542-4aa2-47d6-9d83-f29f584cf771" (UID: "43fdb542-4aa2-47d6-9d83-f29f584cf771"). InnerVolumeSpecName "kube-api-access-67q72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:20.373623 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.373594    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs"
Nov 02 02:01:20.475314 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.471880    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8815f4d6-021e-4c5f-9638-5c53d8470c5d-kube-api-access-9hcbz\" (UniqueName: \"kubernetes.io/projected/8815f4d6-021e-4c5f-9638-5c53d8470c5d-kube-api-access-9hcbz\") pod \"8815f4d6-021e-4c5f-9638-5c53d8470c5d\" (UID: \"8815f4d6-021e-4c5f-9638-5c53d8470c5d\") "
Nov 02 02:01:20.475314 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.472022    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8815f4d6-021e-4c5f-9638-5c53d8470c5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8815f4d6-021e-4c5f-9638-5c53d8470c5d-plugins-config-volume\") pod \"8815f4d6-021e-4c5f-9638-5c53d8470c5d\" (UID: \"8815f4d6-021e-4c5f-9638-5c53d8470c5d\") "
Nov 02 02:01:20.475314 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.473140    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67q72\" (UniqueName: \"kubernetes.io/projected/43fdb542-4aa2-47d6-9d83-f29f584cf771-kube-api-access-67q72\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:20.517359 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.516704    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8815f4d6-021e-4c5f-9638-5c53d8470c5d-plugins-config-volume" pod "8815f4d6-021e-4c5f-9638-5c53d8470c5d" (UID: "8815f4d6-021e-4c5f-9638-5c53d8470c5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:20.604308 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.603865    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8815f4d6-021e-4c5f-9638-5c53d8470c5d-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:20.654310 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.643414    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8815f4d6-021e-4c5f-9638-5c53d8470c5d-kube-api-access-9hcbz" pod "8815f4d6-021e-4c5f-9638-5c53d8470c5d" (UID: "8815f4d6-021e-4c5f-9638-5c53d8470c5d"). InnerVolumeSpecName "kube-api-access-9hcbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:20.654310 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.652508    5639 generic.go:356] "Generic (PLEG): container finished" podID="8815f4d6-021e-4c5f-9638-5c53d8470c5d" containerID="3049b3a58f82c4549afdc6cc77d5b89cebd558dedb9f5d9f34dd6c19116ded74" exitCode=0
Nov 02 02:01:20.654310 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.652674    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs" event={"ID":"8815f4d6-021e-4c5f-9638-5c53d8470c5d","Type":"ContainerDied","Data":"3049b3a58f82c4549afdc6cc77d5b89cebd558dedb9f5d9f34dd6c19116ded74"}
Nov 02 02:01:20.654310 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.652726    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs" event={"ID":"8815f4d6-021e-4c5f-9638-5c53d8470c5d","Type":"ContainerDied","Data":"67420e952da8e0df8f8abe84ac860a7646dab4b7e52fca870f66e8d55698fdf5"}
Nov 02 02:01:20.654310 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.652754    5639 scope.go:122] "RemoveContainer" containerID="3049b3a58f82c4549afdc6cc77d5b89cebd558dedb9f5d9f34dd6c19116ded74"
Nov 02 02:01:20.654310 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.652928    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs"
Nov 02 02:01:20.665528 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.661421    5639 generic.go:356] "Generic (PLEG): container finished" podID="43fdb542-4aa2-47d6-9d83-f29f584cf771" containerID="68ed8a26ab82b43d6a6ab2a3ee684befae8c8a35deb25c862a084f135ab968af" exitCode=0
Nov 02 02:01:20.665528 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.661759    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw" event={"ID":"43fdb542-4aa2-47d6-9d83-f29f584cf771","Type":"ContainerDied","Data":"68ed8a26ab82b43d6a6ab2a3ee684befae8c8a35deb25c862a084f135ab968af"}
Nov 02 02:01:20.665528 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.661836    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw" event={"ID":"43fdb542-4aa2-47d6-9d83-f29f584cf771","Type":"ContainerDied","Data":"70f17dac9218c7e1fef6f832764d4c19a820edcc3a20c1f1299fd2e5f99cd3bb"}
Nov 02 02:01:20.665528 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.661959    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw"
Nov 02 02:01:20.688034 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.688013    5639 scope.go:122] "RemoveContainer" containerID="3049b3a58f82c4549afdc6cc77d5b89cebd558dedb9f5d9f34dd6c19116ded74"
Nov 02 02:01:20.688452 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:01:20.688418    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3049b3a58f82c4549afdc6cc77d5b89cebd558dedb9f5d9f34dd6c19116ded74\": container with ID starting with 3049b3a58f82c4549afdc6cc77d5b89cebd558dedb9f5d9f34dd6c19116ded74 not found: ID does not exist" containerID="3049b3a58f82c4549afdc6cc77d5b89cebd558dedb9f5d9f34dd6c19116ded74"
Nov 02 02:01:20.688594 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.688460    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3049b3a58f82c4549afdc6cc77d5b89cebd558dedb9f5d9f34dd6c19116ded74"} err="failed to get container status \"3049b3a58f82c4549afdc6cc77d5b89cebd558dedb9f5d9f34dd6c19116ded74\": rpc error: code = NotFound desc = could not find container \"3049b3a58f82c4549afdc6cc77d5b89cebd558dedb9f5d9f34dd6c19116ded74\": container with ID starting with 3049b3a58f82c4549afdc6cc77d5b89cebd558dedb9f5d9f34dd6c19116ded74 not found: ID does not exist"
Nov 02 02:01:20.688594 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.688487    5639 scope.go:122] "RemoveContainer" containerID="68ed8a26ab82b43d6a6ab2a3ee684befae8c8a35deb25c862a084f135ab968af"
Nov 02 02:01:20.699786 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.699759    5639 scope.go:122] "RemoveContainer" containerID="68ed8a26ab82b43d6a6ab2a3ee684befae8c8a35deb25c862a084f135ab968af"
Nov 02 02:01:20.700161 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:01:20.700136    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68ed8a26ab82b43d6a6ab2a3ee684befae8c8a35deb25c862a084f135ab968af\": container with ID starting with 68ed8a26ab82b43d6a6ab2a3ee684befae8c8a35deb25c862a084f135ab968af not found: ID does not exist" containerID="68ed8a26ab82b43d6a6ab2a3ee684befae8c8a35deb25c862a084f135ab968af"
Nov 02 02:01:20.700271 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.700169    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68ed8a26ab82b43d6a6ab2a3ee684befae8c8a35deb25c862a084f135ab968af"} err="failed to get container status \"68ed8a26ab82b43d6a6ab2a3ee684befae8c8a35deb25c862a084f135ab968af\": rpc error: code = NotFound desc = could not find container \"68ed8a26ab82b43d6a6ab2a3ee684befae8c8a35deb25c862a084f135ab968af\": container with ID starting with 68ed8a26ab82b43d6a6ab2a3ee684befae8c8a35deb25c862a084f135ab968af not found: ID does not exist"
Nov 02 02:01:20.705881 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.705842    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hcbz\" (UniqueName: \"kubernetes.io/projected/8815f4d6-021e-4c5f-9638-5c53d8470c5d-kube-api-access-9hcbz\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:20.780089 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.780002    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d"]
Nov 02 02:01:20.780741 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.780726    5639 cpu_manager.go:414] "containerMap: removing container" podUID="3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e" containerName="cleanup"
Nov 02 02:01:20.780790 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.780741    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b1f107f-472d-4cfa-9ce8-cc7a0db43d7e" containerName="cleanup"
Nov 02 02:01:20.780790 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.780756    5639 cpu_manager.go:414] "containerMap: removing container" podUID="2c0d592c-c444-4131-ac9c-13e3980e5a6d" containerName="cleanup"
Nov 02 02:01:20.780790 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.780761    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c0d592c-c444-4131-ac9c-13e3980e5a6d" containerName="cleanup"
Nov 02 02:01:20.780790 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.780768    5639 cpu_manager.go:414] "containerMap: removing container" podUID="3fa637e0-ccb8-4713-a16b-f4c353d3cb8f" containerName="cleanup"
Nov 02 02:01:20.780790 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.780772    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa637e0-ccb8-4713-a16b-f4c353d3cb8f" containerName="cleanup"
Nov 02 02:01:20.780790 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.780788    5639 cpu_manager.go:414] "containerMap: removing container" podUID="43fdb542-4aa2-47d6-9d83-f29f584cf771" containerName="payload-processing"
Nov 02 02:01:20.780984 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.780793    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43fdb542-4aa2-47d6-9d83-f29f584cf771" containerName="payload-processing"
Nov 02 02:01:20.780984 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.780816    5639 cpu_manager.go:414] "containerMap: removing container" podUID="3f5659fe-839b-4498-b9eb-2b76d6631417" containerName="cleanup"
Nov 02 02:01:20.780984 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.780820    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f5659fe-839b-4498-b9eb-2b76d6631417" containerName="cleanup"
Nov 02 02:01:20.780984 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.780830    5639 cpu_manager.go:414] "containerMap: removing container" podUID="8815f4d6-021e-4c5f-9638-5c53d8470c5d" containerName="payload-pre-processing"
Nov 02 02:01:20.780984 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.780835    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8815f4d6-021e-4c5f-9638-5c53d8470c5d" containerName="payload-pre-processing"
Nov 02 02:01:20.780984 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.780843    5639 cpu_manager.go:414] "containerMap: removing container" podUID="bccefd52-b090-41c8-88eb-83d79277cd74" containerName="revoke-keys"
Nov 02 02:01:20.780984 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.780848    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bccefd52-b090-41c8-88eb-83d79277cd74" containerName="revoke-keys"
Nov 02 02:01:20.780984 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.780855    5639 cpu_manager.go:414] "containerMap: removing container" podUID="0720184c-4f19-41d9-871f-b3a208201818" containerName="cleanup"
Nov 02 02:01:20.780984 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.780859    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0720184c-4f19-41d9-871f-b3a208201818" containerName="cleanup"
Nov 02 02:01:20.780984 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.780868    5639 cpu_manager.go:414] "containerMap: removing container" podUID="d96d25d2-704b-40d5-9a6b-116c4eb3a4a4" containerName="cleanup"
Nov 02 02:01:20.780984 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.780872    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d96d25d2-704b-40d5-9a6b-116c4eb3a4a4" containerName="cleanup"
Nov 02 02:01:20.780984 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.780892    5639 cpu_manager.go:414] "containerMap: removing container" podUID="2aefa8c8-6056-4fd0-8e70-739dbd421750" containerName="cleanup"
Nov 02 02:01:20.780984 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.780896    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aefa8c8-6056-4fd0-8e70-739dbd421750" containerName="cleanup"
Nov 02 02:01:20.780984 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.780901    5639 cpu_manager.go:414] "containerMap: removing container" podUID="113179bd-c2bc-4fe6-909e-e1387c459bb9" containerName="cleanup"
Nov 02 02:01:20.780984 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.780906    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="113179bd-c2bc-4fe6-909e-e1387c459bb9" containerName="cleanup"
Nov 02 02:01:20.781463 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.781020    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="8815f4d6-021e-4c5f-9638-5c53d8470c5d" containerName="payload-pre-processing"
Nov 02 02:01:20.781463 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.781029    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="bccefd52-b090-41c8-88eb-83d79277cd74" containerName="revoke-keys"
Nov 02 02:01:20.781463 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.781034    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="0720184c-4f19-41d9-871f-b3a208201818" containerName="cleanup"
Nov 02 02:01:20.781463 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.781044    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aefa8c8-6056-4fd0-8e70-739dbd421750" containerName="cleanup"
Nov 02 02:01:20.781463 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.781052    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dabb512-1926-413b-87ab-2a9c87e685f7" containerName="cleanup"
Nov 02 02:01:20.781463 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.781058    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="43fdb542-4aa2-47d6-9d83-f29f584cf771" containerName="payload-processing"
Nov 02 02:01:20.781463 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.781063    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c0d592c-c444-4131-ac9c-13e3980e5a6d" containerName="cleanup"
Nov 02 02:01:20.781463 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.781071    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5b031ba-6e9e-4852-abb4-218ab35f22b9" containerName="cleanup"
Nov 02 02:01:20.781463 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.781077    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="5184916a-2d9b-40cc-9872-9fb0cd055b41" containerName="cleanup"
Nov 02 02:01:20.781463 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.781081    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa637e0-ccb8-4713-a16b-f4c353d3cb8f" containerName="cleanup"
Nov 02 02:01:20.784447 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.784428    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d"
Nov 02 02:01:20.823573 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.823357    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs"]
Nov 02 02:01:20.827472 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.827442    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-218802-694f8f69b7-srcjs"]
Nov 02 02:01:20.838225 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.837703    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:01:20.838485 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.838457    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:01:20.843452 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.843427    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d"]
Nov 02 02:01:20.909495 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.909465    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5mms\" (UniqueName: \"kubernetes.io/projected/c3892edc-da6a-4e5d-b7b6-8edce6f04f4a-kube-api-access-t5mms\") pod \"maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d\" (UID: \"c3892edc-da6a-4e5d-b7b6-8edce6f04f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d"
Nov 02 02:01:20.909667 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.909655    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3892edc-da6a-4e5d-b7b6-8edce6f04f4a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d\" (UID: \"c3892edc-da6a-4e5d-b7b6-8edce6f04f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d"
Nov 02 02:01:20.909742 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.909684    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3892edc-da6a-4e5d-b7b6-8edce6f04f4a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d\" (UID: \"c3892edc-da6a-4e5d-b7b6-8edce6f04f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d"
Nov 02 02:01:20.914552 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.914528    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:01:20.914715 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.914644    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:01:20.937783 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.937752    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw"]
Nov 02 02:01:20.942047 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:20.942022    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-218802-78986f5d89-grjbw"]
Nov 02 02:01:21.011789 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:21.011757    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3892edc-da6a-4e5d-b7b6-8edce6f04f4a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d\" (UID: \"c3892edc-da6a-4e5d-b7b6-8edce6f04f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d"
Nov 02 02:01:21.011789 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:21.011798    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3892edc-da6a-4e5d-b7b6-8edce6f04f4a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d\" (UID: \"c3892edc-da6a-4e5d-b7b6-8edce6f04f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d"
Nov 02 02:01:21.012377 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:21.011985    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5mms\" (UniqueName: \"kubernetes.io/projected/c3892edc-da6a-4e5d-b7b6-8edce6f04f4a-kube-api-access-t5mms\") pod \"maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d\" (UID: \"c3892edc-da6a-4e5d-b7b6-8edce6f04f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d"
Nov 02 02:01:21.012571 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:21.012547    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3892edc-da6a-4e5d-b7b6-8edce6f04f4a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d\" (UID: \"c3892edc-da6a-4e5d-b7b6-8edce6f04f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d"
Nov 02 02:01:21.014675 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:21.014652    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3892edc-da6a-4e5d-b7b6-8edce6f04f4a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d\" (UID: \"c3892edc-da6a-4e5d-b7b6-8edce6f04f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d"
Nov 02 02:01:21.073227 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:21.073160    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5mms\" (UniqueName: \"kubernetes.io/projected/c3892edc-da6a-4e5d-b7b6-8edce6f04f4a-kube-api-access-t5mms\") pod \"maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d\" (UID: \"c3892edc-da6a-4e5d-b7b6-8edce6f04f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d"
Nov 02 02:01:21.098074 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:21.098036    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d"
Nov 02 02:01:21.225764 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:21.225730    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43fdb542-4aa2-47d6-9d83-f29f584cf771" path="/var/lib/kubelet/pods/43fdb542-4aa2-47d6-9d83-f29f584cf771/volumes"
Nov 02 02:01:21.226224 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:21.226206    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8815f4d6-021e-4c5f-9638-5c53d8470c5d" path="/var/lib/kubelet/pods/8815f4d6-021e-4c5f-9638-5c53d8470c5d/volumes"
Nov 02 02:01:21.253683 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:21.253648    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg"]
Nov 02 02:01:21.257559 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:21.257540    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg"
Nov 02 02:01:21.314397 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:21.314358    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg"]
Nov 02 02:01:21.418988 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:21.418944    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40d80410-2d8b-4eb4-b3c3-98b90b3b60b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg\" (UID: \"40d80410-2d8b-4eb4-b3c3-98b90b3b60b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg"
Nov 02 02:01:21.419190 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:21.419146    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99l62\" (UniqueName: \"kubernetes.io/projected/40d80410-2d8b-4eb4-b3c3-98b90b3b60b6-kube-api-access-99l62\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg\" (UID: \"40d80410-2d8b-4eb4-b3c3-98b90b3b60b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg"
Nov 02 02:01:21.419312 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:21.419266    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40d80410-2d8b-4eb4-b3c3-98b90b3b60b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg\" (UID: \"40d80410-2d8b-4eb4-b3c3-98b90b3b60b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg"
Nov 02 02:01:21.492200 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:21.492166    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d"]
Nov 02 02:01:21.492768 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:01:21.492739    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3892edc_da6a_4e5d_b7b6_8edce6f04f4a.slice/crio-8592bbf2f17afdc86bdd7caa1e5cddb7e86d7964138235213552223eddbef6bb WatchSource:0}: Error finding container 8592bbf2f17afdc86bdd7caa1e5cddb7e86d7964138235213552223eddbef6bb: Status 404 returned error can't find the container with id 8592bbf2f17afdc86bdd7caa1e5cddb7e86d7964138235213552223eddbef6bb
Nov 02 02:01:21.520910 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:21.520879    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99l62\" (UniqueName: \"kubernetes.io/projected/40d80410-2d8b-4eb4-b3c3-98b90b3b60b6-kube-api-access-99l62\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg\" (UID: \"40d80410-2d8b-4eb4-b3c3-98b90b3b60b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg"
Nov 02 02:01:21.521072 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:21.520972    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40d80410-2d8b-4eb4-b3c3-98b90b3b60b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg\" (UID: \"40d80410-2d8b-4eb4-b3c3-98b90b3b60b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg"
Nov 02 02:01:21.521148 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:21.521087    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40d80410-2d8b-4eb4-b3c3-98b90b3b60b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg\" (UID: \"40d80410-2d8b-4eb4-b3c3-98b90b3b60b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg"
Nov 02 02:01:21.521790 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:21.521716    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40d80410-2d8b-4eb4-b3c3-98b90b3b60b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg\" (UID: \"40d80410-2d8b-4eb4-b3c3-98b90b3b60b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg"
Nov 02 02:01:21.523487 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:21.523464    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40d80410-2d8b-4eb4-b3c3-98b90b3b60b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg\" (UID: \"40d80410-2d8b-4eb4-b3c3-98b90b3b60b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg"
Nov 02 02:01:21.582035 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:21.581996    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99l62\" (UniqueName: \"kubernetes.io/projected/40d80410-2d8b-4eb4-b3c3-98b90b3b60b6-kube-api-access-99l62\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg\" (UID: \"40d80410-2d8b-4eb4-b3c3-98b90b3b60b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg"
Nov 02 02:01:21.674982 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:21.674945    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d" event={"ID":"c3892edc-da6a-4e5d-b7b6-8edce6f04f4a","Type":"ContainerStarted","Data":"8592bbf2f17afdc86bdd7caa1e5cddb7e86d7964138235213552223eddbef6bb"}
Nov 02 02:01:21.872005 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:21.871974    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg"
Nov 02 02:01:22.252917 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:22.252888    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg"]
Nov 02 02:01:22.254102 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:01:22.254065    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d80410_2d8b_4eb4_b3c3_98b90b3b60b6.slice/crio-5e7242e312cfc99633063ce871b0fef81f011ccae74e63932fdfcdd2cc7f7bf9 WatchSource:0}: Error finding container 5e7242e312cfc99633063ce871b0fef81f011ccae74e63932fdfcdd2cc7f7bf9: Status 404 returned error can't find the container with id 5e7242e312cfc99633063ce871b0fef81f011ccae74e63932fdfcdd2cc7f7bf9
Nov 02 02:01:22.684928 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:22.684892    5639 generic.go:356] "Generic (PLEG): container finished" podID="c3892edc-da6a-4e5d-b7b6-8edce6f04f4a" containerID="bbdc3686ddfbe096ef29e3a4eea470298e991524cbb4e031154daad1706f3ce8" exitCode=0
Nov 02 02:01:22.685087 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:22.684981    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d" event={"ID":"c3892edc-da6a-4e5d-b7b6-8edce6f04f4a","Type":"ContainerDied","Data":"bbdc3686ddfbe096ef29e3a4eea470298e991524cbb4e031154daad1706f3ce8"}
Nov 02 02:01:22.686922 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:22.686893    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg" event={"ID":"40d80410-2d8b-4eb4-b3c3-98b90b3b60b6","Type":"ContainerStarted","Data":"a55b4895ff4b997821a3ab58d9b81775f037ce3f599ec9aa3dfbfbf400bb0821"}
Nov 02 02:01:22.687052 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:22.686950    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg" event={"ID":"40d80410-2d8b-4eb4-b3c3-98b90b3b60b6","Type":"ContainerStarted","Data":"5e7242e312cfc99633063ce871b0fef81f011ccae74e63932fdfcdd2cc7f7bf9"}
Nov 02 02:01:22.919667 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:22.919607    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg" podStartSLOduration=1.919587979 podStartE2EDuration="1.919587979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:01:22.914627557 +0000 UTC m=+2147.818309672" watchObservedRunningTime="2026-11-02 02:01:22.919587979 +0000 UTC m=+2147.823270095"
Nov 02 02:01:23.699071 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:23.699037    5639 generic.go:356] "Generic (PLEG): container finished" podID="40d80410-2d8b-4eb4-b3c3-98b90b3b60b6" containerID="a55b4895ff4b997821a3ab58d9b81775f037ce3f599ec9aa3dfbfbf400bb0821" exitCode=0
Nov 02 02:01:23.699578 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:23.699124    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg" event={"ID":"40d80410-2d8b-4eb4-b3c3-98b90b3b60b6","Type":"ContainerDied","Data":"a55b4895ff4b997821a3ab58d9b81775f037ce3f599ec9aa3dfbfbf400bb0821"}
Nov 02 02:01:23.869781 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:23.869752    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d"
Nov 02 02:01:23.964150 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:23.964061    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3892edc-da6a-4e5d-b7b6-8edce6f04f4a-kube-api-access-t5mms\" (UniqueName: \"kubernetes.io/projected/c3892edc-da6a-4e5d-b7b6-8edce6f04f4a-kube-api-access-t5mms\") pod \"c3892edc-da6a-4e5d-b7b6-8edce6f04f4a\" (UID: \"c3892edc-da6a-4e5d-b7b6-8edce6f04f4a\") "
Nov 02 02:01:23.964150 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:23.964127    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3892edc-da6a-4e5d-b7b6-8edce6f04f4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3892edc-da6a-4e5d-b7b6-8edce6f04f4a-maas-api-openshift-ca\") pod \"c3892edc-da6a-4e5d-b7b6-8edce6f04f4a\" (UID: \"c3892edc-da6a-4e5d-b7b6-8edce6f04f4a\") "
Nov 02 02:01:23.964425 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:23.964316    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3892edc-da6a-4e5d-b7b6-8edce6f04f4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3892edc-da6a-4e5d-b7b6-8edce6f04f4a-maas-api-xks-ca\") pod \"c3892edc-da6a-4e5d-b7b6-8edce6f04f4a\" (UID: \"c3892edc-da6a-4e5d-b7b6-8edce6f04f4a\") "
Nov 02 02:01:23.964547 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:23.964520    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3892edc-da6a-4e5d-b7b6-8edce6f04f4a-maas-api-openshift-ca" pod "c3892edc-da6a-4e5d-b7b6-8edce6f04f4a" (UID: "c3892edc-da6a-4e5d-b7b6-8edce6f04f4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:23.964971 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:23.964947    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3892edc-da6a-4e5d-b7b6-8edce6f04f4a-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:23.966613 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:23.966574    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3892edc-da6a-4e5d-b7b6-8edce6f04f4a-kube-api-access-t5mms" pod "c3892edc-da6a-4e5d-b7b6-8edce6f04f4a" (UID: "c3892edc-da6a-4e5d-b7b6-8edce6f04f4a"). InnerVolumeSpecName "kube-api-access-t5mms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:23.966720 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:23.966700    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3892edc-da6a-4e5d-b7b6-8edce6f04f4a-maas-api-xks-ca" pod "c3892edc-da6a-4e5d-b7b6-8edce6f04f4a" (UID: "c3892edc-da6a-4e5d-b7b6-8edce6f04f4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:01:24.066295 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:24.066230    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5mms\" (UniqueName: \"kubernetes.io/projected/c3892edc-da6a-4e5d-b7b6-8edce6f04f4a-kube-api-access-t5mms\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:24.066295 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:24.066270    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3892edc-da6a-4e5d-b7b6-8edce6f04f4a-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:24.708748 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:24.708714    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d"
Nov 02 02:01:24.708748 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:24.708736    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d" event={"ID":"c3892edc-da6a-4e5d-b7b6-8edce6f04f4a","Type":"ContainerDied","Data":"8592bbf2f17afdc86bdd7caa1e5cddb7e86d7964138235213552223eddbef6bb"}
Nov 02 02:01:24.709252 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:24.708785    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8592bbf2f17afdc86bdd7caa1e5cddb7e86d7964138235213552223eddbef6bb"
Nov 02 02:01:24.879912 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:24.879885    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg"
Nov 02 02:01:24.979139 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:24.979058    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40d80410-2d8b-4eb4-b3c3-98b90b3b60b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40d80410-2d8b-4eb4-b3c3-98b90b3b60b6-maas-api-openshift-ca\") pod \"40d80410-2d8b-4eb4-b3c3-98b90b3b60b6\" (UID: \"40d80410-2d8b-4eb4-b3c3-98b90b3b60b6\") "
Nov 02 02:01:24.979316 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:24.979199    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d80410-2d8b-4eb4-b3c3-98b90b3b60b6-kube-api-access-99l62\" (UniqueName: \"kubernetes.io/projected/40d80410-2d8b-4eb4-b3c3-98b90b3b60b6-kube-api-access-99l62\") pod \"40d80410-2d8b-4eb4-b3c3-98b90b3b60b6\" (UID: \"40d80410-2d8b-4eb4-b3c3-98b90b3b60b6\") "
Nov 02 02:01:24.979374 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:24.979308    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40d80410-2d8b-4eb4-b3c3-98b90b3b60b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40d80410-2d8b-4eb4-b3c3-98b90b3b60b6-maas-api-xks-ca\") pod \"40d80410-2d8b-4eb4-b3c3-98b90b3b60b6\" (UID: \"40d80410-2d8b-4eb4-b3c3-98b90b3b60b6\") "
Nov 02 02:01:24.979537 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:24.979518    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d80410-2d8b-4eb4-b3c3-98b90b3b60b6-maas-api-openshift-ca" pod "40d80410-2d8b-4eb4-b3c3-98b90b3b60b6" (UID: "40d80410-2d8b-4eb4-b3c3-98b90b3b60b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:24.979980 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:24.979957    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40d80410-2d8b-4eb4-b3c3-98b90b3b60b6-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:24.981461 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:24.981437    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40d80410-2d8b-4eb4-b3c3-98b90b3b60b6-maas-api-xks-ca" pod "40d80410-2d8b-4eb4-b3c3-98b90b3b60b6" (UID: "40d80410-2d8b-4eb4-b3c3-98b90b3b60b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:01:24.981580 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:24.981481    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d80410-2d8b-4eb4-b3c3-98b90b3b60b6-kube-api-access-99l62" pod "40d80410-2d8b-4eb4-b3c3-98b90b3b60b6" (UID: "40d80410-2d8b-4eb4-b3c3-98b90b3b60b6"). InnerVolumeSpecName "kube-api-access-99l62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:25.081167 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:25.081135    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40d80410-2d8b-4eb4-b3c3-98b90b3b60b6-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:25.081167 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:25.081164    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99l62\" (UniqueName: \"kubernetes.io/projected/40d80410-2d8b-4eb4-b3c3-98b90b3b60b6-kube-api-access-99l62\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:25.717815 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:25.717786    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg"
Nov 02 02:01:25.718224 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:25.717822    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg" event={"ID":"40d80410-2d8b-4eb4-b3c3-98b90b3b60b6","Type":"ContainerDied","Data":"5e7242e312cfc99633063ce871b0fef81f011ccae74e63932fdfcdd2cc7f7bf9"}
Nov 02 02:01:25.718224 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:25.717872    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e7242e312cfc99633063ce871b0fef81f011ccae74e63932fdfcdd2cc7f7bf9"
Nov 02 02:01:27.423120 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:01:27.423072    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:01:32.330595 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:32.330554    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv"]
Nov 02 02:01:32.436966 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:32.436930    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz"]
Nov 02 02:01:33.626121 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:33.626085    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5"]
Nov 02 02:01:33.683940 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:33.683909    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk"]
Nov 02 02:01:37.330909 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.330855    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv" podUID="6487b7f6-974c-4efb-8acb-37e65e050562" containerName="payload-processing" containerID="cri-o://f9cd24805ae21a14be05b3dc4764b3dfdeb02745eab02e613c823368ee5e6fb5" gracePeriod=25
Nov 02 02:01:37.437519 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.437481    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz" podUID="111a0595-8aab-4dff-acc4-37b21384f047" containerName="payload-pre-processing" containerID="cri-o://9c121bb823e19f68ef64404df64f57491037bfa0120b5dcb2f0921ea3e4ae6b5" gracePeriod=25
Nov 02 02:01:37.487702 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.487668    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d"]
Nov 02 02:01:37.538014 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.537976    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c600fc-mbz8d"]
Nov 02 02:01:37.629495 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.629467    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv"
Nov 02 02:01:37.722945 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.722908    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6487b7f6-974c-4efb-8acb-37e65e050562-kube-api-access-t5fl6\" (UniqueName: \"kubernetes.io/projected/6487b7f6-974c-4efb-8acb-37e65e050562-kube-api-access-t5fl6\") pod \"6487b7f6-974c-4efb-8acb-37e65e050562\" (UID: \"6487b7f6-974c-4efb-8acb-37e65e050562\") "
Nov 02 02:01:37.723147 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.723037    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6487b7f6-974c-4efb-8acb-37e65e050562-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6487b7f6-974c-4efb-8acb-37e65e050562-plugins-config-volume\") pod \"6487b7f6-974c-4efb-8acb-37e65e050562\" (UID: \"6487b7f6-974c-4efb-8acb-37e65e050562\") "
Nov 02 02:01:37.723566 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.723514    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6487b7f6-974c-4efb-8acb-37e65e050562-plugins-config-volume" pod "6487b7f6-974c-4efb-8acb-37e65e050562" (UID: "6487b7f6-974c-4efb-8acb-37e65e050562"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:37.725593 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.725567    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6487b7f6-974c-4efb-8acb-37e65e050562-kube-api-access-t5fl6" pod "6487b7f6-974c-4efb-8acb-37e65e050562" (UID: "6487b7f6-974c-4efb-8acb-37e65e050562"). InnerVolumeSpecName "kube-api-access-t5fl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:37.752332 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.752304    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz"
Nov 02 02:01:37.819012 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.818976    5639 generic.go:356] "Generic (PLEG): container finished" podID="111a0595-8aab-4dff-acc4-37b21384f047" containerID="9c121bb823e19f68ef64404df64f57491037bfa0120b5dcb2f0921ea3e4ae6b5" exitCode=0
Nov 02 02:01:37.819200 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.819026    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz"
Nov 02 02:01:37.819200 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.819059    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz" event={"ID":"111a0595-8aab-4dff-acc4-37b21384f047","Type":"ContainerDied","Data":"9c121bb823e19f68ef64404df64f57491037bfa0120b5dcb2f0921ea3e4ae6b5"}
Nov 02 02:01:37.819200 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.819124    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz" event={"ID":"111a0595-8aab-4dff-acc4-37b21384f047","Type":"ContainerDied","Data":"d5c9f7def9c665615134ac99ff0588999def1e3bde998ca5f22ac04fd45d87f1"}
Nov 02 02:01:37.819200 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.819154    5639 scope.go:122] "RemoveContainer" containerID="9c121bb823e19f68ef64404df64f57491037bfa0120b5dcb2f0921ea3e4ae6b5"
Nov 02 02:01:37.820811 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.820778    5639 generic.go:356] "Generic (PLEG): container finished" podID="6487b7f6-974c-4efb-8acb-37e65e050562" containerID="f9cd24805ae21a14be05b3dc4764b3dfdeb02745eab02e613c823368ee5e6fb5" exitCode=0
Nov 02 02:01:37.820924 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.820810    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv" event={"ID":"6487b7f6-974c-4efb-8acb-37e65e050562","Type":"ContainerDied","Data":"f9cd24805ae21a14be05b3dc4764b3dfdeb02745eab02e613c823368ee5e6fb5"}
Nov 02 02:01:37.820924 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.820834    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv"
Nov 02 02:01:37.820924 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.820846    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv" event={"ID":"6487b7f6-974c-4efb-8acb-37e65e050562","Type":"ContainerDied","Data":"4f2d319f83a1949ad0f5dcdc6484826814766b5c2564e127c4cc688a41744a47"}
Nov 02 02:01:37.824101 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.824082    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/111a0595-8aab-4dff-acc4-37b21384f047-kube-api-access-bpf8h\" (UniqueName: \"kubernetes.io/projected/111a0595-8aab-4dff-acc4-37b21384f047-kube-api-access-bpf8h\") pod \"111a0595-8aab-4dff-acc4-37b21384f047\" (UID: \"111a0595-8aab-4dff-acc4-37b21384f047\") "
Nov 02 02:01:37.824435 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.824399    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/111a0595-8aab-4dff-acc4-37b21384f047-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111a0595-8aab-4dff-acc4-37b21384f047-plugins-config-volume\") pod \"111a0595-8aab-4dff-acc4-37b21384f047\" (UID: \"111a0595-8aab-4dff-acc4-37b21384f047\") "
Nov 02 02:01:37.824954 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.824802    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/111a0595-8aab-4dff-acc4-37b21384f047-plugins-config-volume" pod "111a0595-8aab-4dff-acc4-37b21384f047" (UID: "111a0595-8aab-4dff-acc4-37b21384f047"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:37.825138 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.825121    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5fl6\" (UniqueName: \"kubernetes.io/projected/6487b7f6-974c-4efb-8acb-37e65e050562-kube-api-access-t5fl6\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:37.825190 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.825148    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6487b7f6-974c-4efb-8acb-37e65e050562-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:37.825190 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.825164    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111a0595-8aab-4dff-acc4-37b21384f047-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:37.826512 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.826491    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111a0595-8aab-4dff-acc4-37b21384f047-kube-api-access-bpf8h" pod "111a0595-8aab-4dff-acc4-37b21384f047" (UID: "111a0595-8aab-4dff-acc4-37b21384f047"). InnerVolumeSpecName "kube-api-access-bpf8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:37.830709 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.830688    5639 scope.go:122] "RemoveContainer" containerID="9c121bb823e19f68ef64404df64f57491037bfa0120b5dcb2f0921ea3e4ae6b5"
Nov 02 02:01:37.831072 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:01:37.831025    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c121bb823e19f68ef64404df64f57491037bfa0120b5dcb2f0921ea3e4ae6b5\": container with ID starting with 9c121bb823e19f68ef64404df64f57491037bfa0120b5dcb2f0921ea3e4ae6b5 not found: ID does not exist" containerID="9c121bb823e19f68ef64404df64f57491037bfa0120b5dcb2f0921ea3e4ae6b5"
Nov 02 02:01:37.831072 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.831060    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c121bb823e19f68ef64404df64f57491037bfa0120b5dcb2f0921ea3e4ae6b5"} err="failed to get container status \"9c121bb823e19f68ef64404df64f57491037bfa0120b5dcb2f0921ea3e4ae6b5\": rpc error: code = NotFound desc = could not find container \"9c121bb823e19f68ef64404df64f57491037bfa0120b5dcb2f0921ea3e4ae6b5\": container with ID starting with 9c121bb823e19f68ef64404df64f57491037bfa0120b5dcb2f0921ea3e4ae6b5 not found: ID does not exist"
Nov 02 02:01:37.831166 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.831078    5639 scope.go:122] "RemoveContainer" containerID="f9cd24805ae21a14be05b3dc4764b3dfdeb02745eab02e613c823368ee5e6fb5"
Nov 02 02:01:37.842119 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.842087    5639 scope.go:122] "RemoveContainer" containerID="f9cd24805ae21a14be05b3dc4764b3dfdeb02745eab02e613c823368ee5e6fb5"
Nov 02 02:01:37.842468 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:01:37.842445    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9cd24805ae21a14be05b3dc4764b3dfdeb02745eab02e613c823368ee5e6fb5\": container with ID starting with f9cd24805ae21a14be05b3dc4764b3dfdeb02745eab02e613c823368ee5e6fb5 not found: ID does not exist" containerID="f9cd24805ae21a14be05b3dc4764b3dfdeb02745eab02e613c823368ee5e6fb5"
Nov 02 02:01:37.842578 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.842474    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9cd24805ae21a14be05b3dc4764b3dfdeb02745eab02e613c823368ee5e6fb5"} err="failed to get container status \"f9cd24805ae21a14be05b3dc4764b3dfdeb02745eab02e613c823368ee5e6fb5\": rpc error: code = NotFound desc = could not find container \"f9cd24805ae21a14be05b3dc4764b3dfdeb02745eab02e613c823368ee5e6fb5\": container with ID starting with f9cd24805ae21a14be05b3dc4764b3dfdeb02745eab02e613c823368ee5e6fb5 not found: ID does not exist"
Nov 02 02:01:37.926459 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:37.926421    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpf8h\" (UniqueName: \"kubernetes.io/projected/111a0595-8aab-4dff-acc4-37b21384f047-kube-api-access-bpf8h\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:38.026702 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:38.026667    5639 scope.go:122] "RemoveContainer" containerID="885fe77a7882c680929f4d1381c8c1fc8b63a44a0d942b9733ec814a0711222d"
Nov 02 02:01:38.040062 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:38.040029    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv"]
Nov 02 02:01:38.049603 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:38.049563    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-19436b-bd77f9c4f-z72pv"]
Nov 02 02:01:38.238146 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:38.238105    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz"]
Nov 02 02:01:38.243642 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:38.243616    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-19436b-5956b65b79-pv6wz"]
Nov 02 02:01:38.627337 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:38.627220    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5" podUID="1f9e832d-76eb-41c1-91b9-db784e55c47e" containerName="payload-processing" containerID="cri-o://2bbfa81577528780b3d5cb20e0c63f6ef3a81d622cea2094b420903a1706d67f" gracePeriod=25
Nov 02 02:01:38.684732 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:38.684651    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk" podUID="05e9bda3-da6b-437a-9b42-3a9a9a4aafcd" containerName="payload-pre-processing" containerID="cri-o://f09bb8eb4648d9c8e830d25a7f36dbec1ed0702c35cc6a675c1d8fe564e41101" gracePeriod=25
Nov 02 02:01:38.831759 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:38.831724    5639 generic.go:356] "Generic (PLEG): container finished" podID="05e9bda3-da6b-437a-9b42-3a9a9a4aafcd" containerID="f09bb8eb4648d9c8e830d25a7f36dbec1ed0702c35cc6a675c1d8fe564e41101" exitCode=0
Nov 02 02:01:38.831972 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:38.831800    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk" event={"ID":"05e9bda3-da6b-437a-9b42-3a9a9a4aafcd","Type":"ContainerDied","Data":"f09bb8eb4648d9c8e830d25a7f36dbec1ed0702c35cc6a675c1d8fe564e41101"}
Nov 02 02:01:38.833626 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:38.833603    5639 generic.go:356] "Generic (PLEG): container finished" podID="1f9e832d-76eb-41c1-91b9-db784e55c47e" containerID="2bbfa81577528780b3d5cb20e0c63f6ef3a81d622cea2094b420903a1706d67f" exitCode=0
Nov 02 02:01:38.833731 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:38.833663    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5" event={"ID":"1f9e832d-76eb-41c1-91b9-db784e55c47e","Type":"ContainerDied","Data":"2bbfa81577528780b3d5cb20e0c63f6ef3a81d622cea2094b420903a1706d67f"}
Nov 02 02:01:38.985744 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:38.985713    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5"
Nov 02 02:01:39.020009 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:39.019980    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk"
Nov 02 02:01:39.145299 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:39.144656    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05e9bda3-da6b-437a-9b42-3a9a9a4aafcd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05e9bda3-da6b-437a-9b42-3a9a9a4aafcd-plugins-config-volume\") pod \"05e9bda3-da6b-437a-9b42-3a9a9a4aafcd\" (UID: \"05e9bda3-da6b-437a-9b42-3a9a9a4aafcd\") "
Nov 02 02:01:39.145299 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:39.144847    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f9e832d-76eb-41c1-91b9-db784e55c47e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f9e832d-76eb-41c1-91b9-db784e55c47e-plugins-config-volume\") pod \"1f9e832d-76eb-41c1-91b9-db784e55c47e\" (UID: \"1f9e832d-76eb-41c1-91b9-db784e55c47e\") "
Nov 02 02:01:39.145299 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:39.144943    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f9e832d-76eb-41c1-91b9-db784e55c47e-kube-api-access-n9rww\" (UniqueName: \"kubernetes.io/projected/1f9e832d-76eb-41c1-91b9-db784e55c47e-kube-api-access-n9rww\") pod \"1f9e832d-76eb-41c1-91b9-db784e55c47e\" (UID: \"1f9e832d-76eb-41c1-91b9-db784e55c47e\") "
Nov 02 02:01:39.145299 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:39.144976    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05e9bda3-da6b-437a-9b42-3a9a9a4aafcd-kube-api-access-sbbvg\" (UniqueName: \"kubernetes.io/projected/05e9bda3-da6b-437a-9b42-3a9a9a4aafcd-kube-api-access-sbbvg\") pod \"05e9bda3-da6b-437a-9b42-3a9a9a4aafcd\" (UID: \"05e9bda3-da6b-437a-9b42-3a9a9a4aafcd\") "
Nov 02 02:01:39.149292 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:39.145692    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f9e832d-76eb-41c1-91b9-db784e55c47e-plugins-config-volume" pod "1f9e832d-76eb-41c1-91b9-db784e55c47e" (UID: "1f9e832d-76eb-41c1-91b9-db784e55c47e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:39.149292 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:39.146420    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f9e832d-76eb-41c1-91b9-db784e55c47e-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:39.158942 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:39.151618    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05e9bda3-da6b-437a-9b42-3a9a9a4aafcd-plugins-config-volume" pod "05e9bda3-da6b-437a-9b42-3a9a9a4aafcd" (UID: "05e9bda3-da6b-437a-9b42-3a9a9a4aafcd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:39.158942 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:39.154609    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f9e832d-76eb-41c1-91b9-db784e55c47e-kube-api-access-n9rww" pod "1f9e832d-76eb-41c1-91b9-db784e55c47e" (UID: "1f9e832d-76eb-41c1-91b9-db784e55c47e"). InnerVolumeSpecName "kube-api-access-n9rww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:39.158942 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:39.154696    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e9bda3-da6b-437a-9b42-3a9a9a4aafcd-kube-api-access-sbbvg" pod "05e9bda3-da6b-437a-9b42-3a9a9a4aafcd" (UID: "05e9bda3-da6b-437a-9b42-3a9a9a4aafcd"). InnerVolumeSpecName "kube-api-access-sbbvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:39.221487 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:39.221321    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="111a0595-8aab-4dff-acc4-37b21384f047" path="/var/lib/kubelet/pods/111a0595-8aab-4dff-acc4-37b21384f047/volumes"
Nov 02 02:01:39.222028 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:39.221997    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6487b7f6-974c-4efb-8acb-37e65e050562" path="/var/lib/kubelet/pods/6487b7f6-974c-4efb-8acb-37e65e050562/volumes"
Nov 02 02:01:39.222585 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:39.222564    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3892edc-da6a-4e5d-b7b6-8edce6f04f4a" path="/var/lib/kubelet/pods/c3892edc-da6a-4e5d-b7b6-8edce6f04f4a/volumes"
Nov 02 02:01:39.248326 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:39.248294    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05e9bda3-da6b-437a-9b42-3a9a9a4aafcd-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:39.248326 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:39.248322    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9rww\" (UniqueName: \"kubernetes.io/projected/1f9e832d-76eb-41c1-91b9-db784e55c47e-kube-api-access-n9rww\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:39.248326 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:39.248332    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbbvg\" (UniqueName: \"kubernetes.io/projected/05e9bda3-da6b-437a-9b42-3a9a9a4aafcd-kube-api-access-sbbvg\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:39.842510 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:39.842459    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk" event={"ID":"05e9bda3-da6b-437a-9b42-3a9a9a4aafcd","Type":"ContainerDied","Data":"93e70366ba65f96d327e92b1de4c4a1c7922c9e5e9e891f78f0b7a24a317e5b9"}
Nov 02 02:01:39.843003 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:39.842550    5639 scope.go:122] "RemoveContainer" containerID="f09bb8eb4648d9c8e830d25a7f36dbec1ed0702c35cc6a675c1d8fe564e41101"
Nov 02 02:01:39.843003 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:39.842491    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk"
Nov 02 02:01:39.844348 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:39.844266    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5"
Nov 02 02:01:39.844442 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:39.844349    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5" event={"ID":"1f9e832d-76eb-41c1-91b9-db784e55c47e","Type":"ContainerDied","Data":"50f29fc49d204725524827acecfee66f97af1d6d7bb19686f029ffa09451c109"}
Nov 02 02:01:39.854150 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:39.854128    5639 scope.go:122] "RemoveContainer" containerID="2bbfa81577528780b3d5cb20e0c63f6ef3a81d622cea2094b420903a1706d67f"
Nov 02 02:01:40.010676 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:40.010633    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5"]
Nov 02 02:01:40.015228 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:40.015194    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c600fc-c9f9d4cb9-kfqw5"]
Nov 02 02:01:40.124264 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:40.124144    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk"]
Nov 02 02:01:40.127998 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:40.127969    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c600fc-6f98f746cf-h6ngk"]
Nov 02 02:01:41.220612 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:41.220580    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05e9bda3-da6b-437a-9b42-3a9a9a4aafcd" path="/var/lib/kubelet/pods/05e9bda3-da6b-437a-9b42-3a9a9a4aafcd/volumes"
Nov 02 02:01:41.221107 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:41.221065    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f9e832d-76eb-41c1-91b9-db784e55c47e" path="/var/lib/kubelet/pods/1f9e832d-76eb-41c1-91b9-db784e55c47e/volumes"
Nov 02 02:01:44.456606 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:44.456566    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"]
Nov 02 02:01:44.457046 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:44.456786    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw" podUID="25cd82fe-a02f-45cf-9a41-bdbaea642b30" containerName="istio-proxy" containerID="cri-o://e12caf8cd904f257f4df4da84b5ed8b236f6e573f0bbccc85f37697b9b94c095" gracePeriod=30
Nov 02 02:01:47.285363 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.284709    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-109b07-q2hk7"]
Nov 02 02:01:47.332993 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.332956    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-109b07-q2hk7"]
Nov 02 02:01:47.663668 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.663625    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r"]
Nov 02 02:01:47.664429 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.664412    5639 cpu_manager.go:414] "containerMap: removing container" podUID="40d80410-2d8b-4eb4-b3c3-98b90b3b60b6" containerName="revoke-keys"
Nov 02 02:01:47.664495 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.664430    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d80410-2d8b-4eb4-b3c3-98b90b3b60b6" containerName="revoke-keys"
Nov 02 02:01:47.664495 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.664455    5639 cpu_manager.go:414] "containerMap: removing container" podUID="6487b7f6-974c-4efb-8acb-37e65e050562" containerName="payload-processing"
Nov 02 02:01:47.664495 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.664460    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6487b7f6-974c-4efb-8acb-37e65e050562" containerName="payload-processing"
Nov 02 02:01:47.664495 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.664473    5639 cpu_manager.go:414] "containerMap: removing container" podUID="05e9bda3-da6b-437a-9b42-3a9a9a4aafcd" containerName="payload-pre-processing"
Nov 02 02:01:47.664495 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.664478    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05e9bda3-da6b-437a-9b42-3a9a9a4aafcd" containerName="payload-pre-processing"
Nov 02 02:01:47.664495 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.664490    5639 cpu_manager.go:414] "containerMap: removing container" podUID="1f9e832d-76eb-41c1-91b9-db784e55c47e" containerName="payload-processing"
Nov 02 02:01:47.664495 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.664495    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f9e832d-76eb-41c1-91b9-db784e55c47e" containerName="payload-processing"
Nov 02 02:01:47.664741 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.664510    5639 cpu_manager.go:414] "containerMap: removing container" podUID="c3892edc-da6a-4e5d-b7b6-8edce6f04f4a" containerName="revoke-keys"
Nov 02 02:01:47.664741 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.664515    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3892edc-da6a-4e5d-b7b6-8edce6f04f4a" containerName="revoke-keys"
Nov 02 02:01:47.664741 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.664531    5639 cpu_manager.go:414] "containerMap: removing container" podUID="111a0595-8aab-4dff-acc4-37b21384f047" containerName="payload-pre-processing"
Nov 02 02:01:47.664741 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.664535    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="111a0595-8aab-4dff-acc4-37b21384f047" containerName="payload-pre-processing"
Nov 02 02:01:47.664741 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.664633    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d80410-2d8b-4eb4-b3c3-98b90b3b60b6" containerName="revoke-keys"
Nov 02 02:01:47.664741 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.664643    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="6487b7f6-974c-4efb-8acb-37e65e050562" containerName="payload-processing"
Nov 02 02:01:47.664741 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.664650    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="05e9bda3-da6b-437a-9b42-3a9a9a4aafcd" containerName="payload-pre-processing"
Nov 02 02:01:47.664741 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.664657    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f9e832d-76eb-41c1-91b9-db784e55c47e" containerName="payload-processing"
Nov 02 02:01:47.664741 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.664665    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3892edc-da6a-4e5d-b7b6-8edce6f04f4a" containerName="revoke-keys"
Nov 02 02:01:47.664741 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.664673    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="111a0595-8aab-4dff-acc4-37b21384f047" containerName="payload-pre-processing"
Nov 02 02:01:47.669967 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.669938    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r"
Nov 02 02:01:47.722723 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.722682    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:01:47.723525 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.723501    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:01:47.725171 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.725144    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r"]
Nov 02 02:01:47.769966 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.769928    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:01:47.769966 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.769961    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:01:47.798664 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.798618    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7txv4\" (UniqueName: \"kubernetes.io/projected/babf6f72-f00a-45eb-8295-1054a1d61850-kube-api-access-7txv4\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r\" (UID: \"babf6f72-f00a-45eb-8295-1054a1d61850\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r"
Nov 02 02:01:47.798971 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.798948    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babf6f72-f00a-45eb-8295-1054a1d61850-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r\" (UID: \"babf6f72-f00a-45eb-8295-1054a1d61850\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r"
Nov 02 02:01:47.799027 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.798991    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babf6f72-f00a-45eb-8295-1054a1d61850-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r\" (UID: \"babf6f72-f00a-45eb-8295-1054a1d61850\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r"
Nov 02 02:01:47.900175 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.900135    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babf6f72-f00a-45eb-8295-1054a1d61850-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r\" (UID: \"babf6f72-f00a-45eb-8295-1054a1d61850\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r"
Nov 02 02:01:47.900445 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.900187    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babf6f72-f00a-45eb-8295-1054a1d61850-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r\" (UID: \"babf6f72-f00a-45eb-8295-1054a1d61850\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r"
Nov 02 02:01:47.900445 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.900263    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7txv4\" (UniqueName: \"kubernetes.io/projected/babf6f72-f00a-45eb-8295-1054a1d61850-kube-api-access-7txv4\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r\" (UID: \"babf6f72-f00a-45eb-8295-1054a1d61850\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r"
Nov 02 02:01:47.900965 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.900936    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babf6f72-f00a-45eb-8295-1054a1d61850-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r\" (UID: \"babf6f72-f00a-45eb-8295-1054a1d61850\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r"
Nov 02 02:01:47.902786 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.902763    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babf6f72-f00a-45eb-8295-1054a1d61850-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r\" (UID: \"babf6f72-f00a-45eb-8295-1054a1d61850\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r"
Nov 02 02:01:47.961455 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.961376    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7txv4\" (UniqueName: \"kubernetes.io/projected/babf6f72-f00a-45eb-8295-1054a1d61850-kube-api-access-7txv4\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r\" (UID: \"babf6f72-f00a-45eb-8295-1054a1d61850\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r"
Nov 02 02:01:47.985473 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:47.985443    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r"
Nov 02 02:01:48.369853 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:48.369821    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r"]
Nov 02 02:01:48.371359 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:01:48.371327    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbabf6f72_f00a_45eb_8295_1054a1d61850.slice/crio-71271e6928ab9b4996857e8ea4fc7a93c5df73e1a410c9c03f5ea71b0d591c7a WatchSource:0}: Error finding container 71271e6928ab9b4996857e8ea4fc7a93c5df73e1a410c9c03f5ea71b0d591c7a: Status 404 returned error can't find the container with id 71271e6928ab9b4996857e8ea4fc7a93c5df73e1a410c9c03f5ea71b0d591c7a
Nov 02 02:01:48.745904 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:48.745866    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk"]
Nov 02 02:01:48.795954 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:48.795919    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-zrrnk"]
Nov 02 02:01:48.998008 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:48.997914    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r" event={"ID":"babf6f72-f00a-45eb-8295-1054a1d61850","Type":"ContainerStarted","Data":"c855820ba3da811f33f8bff6afd3aaf04d22b6e9980eae77a831031e6b5b63de"}
Nov 02 02:01:48.998008 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:48.997975    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r" event={"ID":"babf6f72-f00a-45eb-8295-1054a1d61850","Type":"ContainerStarted","Data":"71271e6928ab9b4996857e8ea4fc7a93c5df73e1a410c9c03f5ea71b0d591c7a"}
Nov 02 02:01:49.114161 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.114110    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r" podStartSLOduration=2.114095886 podStartE2EDuration="2.114095886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:01:49.112958896 +0000 UTC m=+2174.016641010" watchObservedRunningTime="2026-11-02 02:01:49.114095886 +0000 UTC m=+2174.017778001"
Nov 02 02:01:49.220515 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.220479    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b605a17f-6063-48fd-bc46-19de1a0ed9ef" path="/var/lib/kubelet/pods/b605a17f-6063-48fd-bc46-19de1a0ed9ef/volumes"
Nov 02 02:01:49.220992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.220976    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba22028e-a74a-4b4c-baef-dd9502007685" path="/var/lib/kubelet/pods/ba22028e-a74a-4b4c-baef-dd9502007685/volumes"
Nov 02 02:01:49.736328 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.736300    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 02:01:49.833947 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.833911    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-workload-certs\") pod \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") "
Nov 02 02:01:49.834143 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.833959    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-token\" (UniqueName: \"kubernetes.io/projected/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-token\") pod \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") "
Nov 02 02:01:49.834143 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.833979    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-workload-socket\") pod \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") "
Nov 02 02:01:49.834143 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.834016    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25cd82fe-a02f-45cf-9a41-bdbaea642b30-kube-api-access-xjln4\" (UniqueName: \"kubernetes.io/projected/25cd82fe-a02f-45cf-9a41-bdbaea642b30-kube-api-access-xjln4\") pod \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") "
Nov 02 02:01:49.834143 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.834050    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-credential-socket\") pod \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") "
Nov 02 02:01:49.834143 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.834107    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-data\") pod \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") "
Nov 02 02:01:49.834143 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.834141    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istiod-ca-cert\") pod \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") "
Nov 02 02:01:49.834546 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.834185    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-envoy\") pod \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") "
Nov 02 02:01:49.834546 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.834227    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-workload-certs" pod "25cd82fe-a02f-45cf-9a41-bdbaea642b30" (UID: "25cd82fe-a02f-45cf-9a41-bdbaea642b30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:49.834546 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.834334    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-podinfo\") pod \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\" (UID: \"25cd82fe-a02f-45cf-9a41-bdbaea642b30\") "
Nov 02 02:01:49.834546 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.834398    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-credential-socket" pod "25cd82fe-a02f-45cf-9a41-bdbaea642b30" (UID: "25cd82fe-a02f-45cf-9a41-bdbaea642b30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:49.834546 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.834416    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-workload-socket" pod "25cd82fe-a02f-45cf-9a41-bdbaea642b30" (UID: "25cd82fe-a02f-45cf-9a41-bdbaea642b30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:49.834834 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.834557    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-data" pod "25cd82fe-a02f-45cf-9a41-bdbaea642b30" (UID: "25cd82fe-a02f-45cf-9a41-bdbaea642b30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:49.834834 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.834627    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istiod-ca-cert" pod "25cd82fe-a02f-45cf-9a41-bdbaea642b30" (UID: "25cd82fe-a02f-45cf-9a41-bdbaea642b30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:49.835245 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.835223    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:49.835373 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.835253    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:49.835373 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.835269    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:49.835373 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.835303    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:49.835373 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.835318    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:49.836589 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.836562    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-envoy" pod "25cd82fe-a02f-45cf-9a41-bdbaea642b30" (UID: "25cd82fe-a02f-45cf-9a41-bdbaea642b30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:49.836589 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.836580    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25cd82fe-a02f-45cf-9a41-bdbaea642b30-kube-api-access-xjln4" pod "25cd82fe-a02f-45cf-9a41-bdbaea642b30" (UID: "25cd82fe-a02f-45cf-9a41-bdbaea642b30"). InnerVolumeSpecName "kube-api-access-xjln4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:49.836589 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.836566    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-token" pod "25cd82fe-a02f-45cf-9a41-bdbaea642b30" (UID: "25cd82fe-a02f-45cf-9a41-bdbaea642b30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:49.836589 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.836581    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-podinfo" pod "25cd82fe-a02f-45cf-9a41-bdbaea642b30" (UID: "25cd82fe-a02f-45cf-9a41-bdbaea642b30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:01:49.937108 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.937075    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:49.937108 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.937110    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:49.937411 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.937126    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjln4\" (UniqueName: \"kubernetes.io/projected/25cd82fe-a02f-45cf-9a41-bdbaea642b30-kube-api-access-xjln4\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:49.937411 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:49.937141    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25cd82fe-a02f-45cf-9a41-bdbaea642b30-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:50.007051 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:50.007013    5639 generic.go:356] "Generic (PLEG): container finished" podID="babf6f72-f00a-45eb-8295-1054a1d61850" containerID="c855820ba3da811f33f8bff6afd3aaf04d22b6e9980eae77a831031e6b5b63de" exitCode=0
Nov 02 02:01:50.007236 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:50.007047    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r" event={"ID":"babf6f72-f00a-45eb-8295-1054a1d61850","Type":"ContainerDied","Data":"c855820ba3da811f33f8bff6afd3aaf04d22b6e9980eae77a831031e6b5b63de"}
Nov 02 02:01:50.010232 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:50.010200    5639 generic.go:356] "Generic (PLEG): container finished" podID="25cd82fe-a02f-45cf-9a41-bdbaea642b30" containerID="e12caf8cd904f257f4df4da84b5ed8b236f6e573f0bbccc85f37697b9b94c095" exitCode=0
Nov 02 02:01:50.010525 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:50.010503    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw" event={"ID":"25cd82fe-a02f-45cf-9a41-bdbaea642b30","Type":"ContainerDied","Data":"e12caf8cd904f257f4df4da84b5ed8b236f6e573f0bbccc85f37697b9b94c095"}
Nov 02 02:01:50.010624 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:50.010608    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw" event={"ID":"25cd82fe-a02f-45cf-9a41-bdbaea642b30","Type":"ContainerDied","Data":"fe8534225f0a4e93b09a0249c5774baa6be3ae163d15732a75024d63c4f85b89"}
Nov 02 02:01:50.010685 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:50.010652    5639 scope.go:122] "RemoveContainer" containerID="e12caf8cd904f257f4df4da84b5ed8b236f6e573f0bbccc85f37697b9b94c095"
Nov 02 02:01:50.010887 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:50.010869    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"
Nov 02 02:01:50.030477 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:50.030457    5639 scope.go:122] "RemoveContainer" containerID="e4a8674d8a142bb9b0d93ad7b91bb3bf25b3066656b2892728e2b62aa4a9ca6f"
Nov 02 02:01:50.040709 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:50.040686    5639 scope.go:122] "RemoveContainer" containerID="e12caf8cd904f257f4df4da84b5ed8b236f6e573f0bbccc85f37697b9b94c095"
Nov 02 02:01:50.041023 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:01:50.040998    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e12caf8cd904f257f4df4da84b5ed8b236f6e573f0bbccc85f37697b9b94c095\": container with ID starting with e12caf8cd904f257f4df4da84b5ed8b236f6e573f0bbccc85f37697b9b94c095 not found: ID does not exist" containerID="e12caf8cd904f257f4df4da84b5ed8b236f6e573f0bbccc85f37697b9b94c095"
Nov 02 02:01:50.041078 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:50.041036    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e12caf8cd904f257f4df4da84b5ed8b236f6e573f0bbccc85f37697b9b94c095"} err="failed to get container status \"e12caf8cd904f257f4df4da84b5ed8b236f6e573f0bbccc85f37697b9b94c095\": rpc error: code = NotFound desc = could not find container \"e12caf8cd904f257f4df4da84b5ed8b236f6e573f0bbccc85f37697b9b94c095\": container with ID starting with e12caf8cd904f257f4df4da84b5ed8b236f6e573f0bbccc85f37697b9b94c095 not found: ID does not exist"
Nov 02 02:01:50.041078 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:50.041061    5639 scope.go:122] "RemoveContainer" containerID="e4a8674d8a142bb9b0d93ad7b91bb3bf25b3066656b2892728e2b62aa4a9ca6f"
Nov 02 02:01:50.041407 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:01:50.041376    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4a8674d8a142bb9b0d93ad7b91bb3bf25b3066656b2892728e2b62aa4a9ca6f\": container with ID starting with e4a8674d8a142bb9b0d93ad7b91bb3bf25b3066656b2892728e2b62aa4a9ca6f not found: ID does not exist" containerID="e4a8674d8a142bb9b0d93ad7b91bb3bf25b3066656b2892728e2b62aa4a9ca6f"
Nov 02 02:01:50.041494 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:50.041416    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4a8674d8a142bb9b0d93ad7b91bb3bf25b3066656b2892728e2b62aa4a9ca6f"} err="failed to get container status \"e4a8674d8a142bb9b0d93ad7b91bb3bf25b3066656b2892728e2b62aa4a9ca6f\": rpc error: code = NotFound desc = could not find container \"e4a8674d8a142bb9b0d93ad7b91bb3bf25b3066656b2892728e2b62aa4a9ca6f\": container with ID starting with e4a8674d8a142bb9b0d93ad7b91bb3bf25b3066656b2892728e2b62aa4a9ca6f not found: ID does not exist"
Nov 02 02:01:50.241162 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:50.240933    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"]
Nov 02 02:01:50.244794 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:50.244768    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c600fc-openshift-default-7ff45454cf-rzptw"]
Nov 02 02:01:51.187159 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:51.187136    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r"
Nov 02 02:01:51.220895 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:51.220863    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25cd82fe-a02f-45cf-9a41-bdbaea642b30" path="/var/lib/kubelet/pods/25cd82fe-a02f-45cf-9a41-bdbaea642b30/volumes"
Nov 02 02:01:51.255798 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:51.255747    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/babf6f72-f00a-45eb-8295-1054a1d61850-kube-api-access-7txv4\" (UniqueName: \"kubernetes.io/projected/babf6f72-f00a-45eb-8295-1054a1d61850-kube-api-access-7txv4\") pod \"babf6f72-f00a-45eb-8295-1054a1d61850\" (UID: \"babf6f72-f00a-45eb-8295-1054a1d61850\") "
Nov 02 02:01:51.255929 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:51.255892    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/babf6f72-f00a-45eb-8295-1054a1d61850-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babf6f72-f00a-45eb-8295-1054a1d61850-maas-api-xks-ca\") pod \"babf6f72-f00a-45eb-8295-1054a1d61850\" (UID: \"babf6f72-f00a-45eb-8295-1054a1d61850\") "
Nov 02 02:01:51.255995 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:51.255947    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/babf6f72-f00a-45eb-8295-1054a1d61850-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babf6f72-f00a-45eb-8295-1054a1d61850-maas-api-openshift-ca\") pod \"babf6f72-f00a-45eb-8295-1054a1d61850\" (UID: \"babf6f72-f00a-45eb-8295-1054a1d61850\") "
Nov 02 02:01:51.256345 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:51.256318    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/babf6f72-f00a-45eb-8295-1054a1d61850-maas-api-openshift-ca" pod "babf6f72-f00a-45eb-8295-1054a1d61850" (UID: "babf6f72-f00a-45eb-8295-1054a1d61850"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:51.256656 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:51.256640    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babf6f72-f00a-45eb-8295-1054a1d61850-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:51.258003 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:51.257977    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/babf6f72-f00a-45eb-8295-1054a1d61850-kube-api-access-7txv4" pod "babf6f72-f00a-45eb-8295-1054a1d61850" (UID: "babf6f72-f00a-45eb-8295-1054a1d61850"). InnerVolumeSpecName "kube-api-access-7txv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:51.258003 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:51.257996    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/babf6f72-f00a-45eb-8295-1054a1d61850-maas-api-xks-ca" pod "babf6f72-f00a-45eb-8295-1054a1d61850" (UID: "babf6f72-f00a-45eb-8295-1054a1d61850"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:01:51.358170 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:51.358140    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7txv4\" (UniqueName: \"kubernetes.io/projected/babf6f72-f00a-45eb-8295-1054a1d61850-kube-api-access-7txv4\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:51.358170 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:51.358170    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babf6f72-f00a-45eb-8295-1054a1d61850-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:52.028607 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:52.028578    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r"
Nov 02 02:01:52.028774 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:52.028614    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r" event={"ID":"babf6f72-f00a-45eb-8295-1054a1d61850","Type":"ContainerDied","Data":"71271e6928ab9b4996857e8ea4fc7a93c5df73e1a410c9c03f5ea71b0d591c7a"}
Nov 02 02:01:52.028774 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:52.028672    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71271e6928ab9b4996857e8ea4fc7a93c5df73e1a410c9c03f5ea71b0d591c7a"
Nov 02 02:01:54.226394 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:54.226344    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"]
Nov 02 02:01:54.226871 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:54.226613    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf" podUID="b2e9988f-011a-4a80-ad65-d54e5266e8dd" containerName="istio-proxy" containerID="cri-o://397a4da7f345c8732a14e16debefca5d88a4f39c3cbc66991ce02c624224df3d" gracePeriod=30
Nov 02 02:01:55.541586 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:55.541543    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:01:57.422463 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:01:57.422426    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:01:57.638359 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:57.638313    5639 patch_prober.go:29] interesting pod/e2e-api-a-109b07-openshift-default-54786bf495-brglf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:01:57.638534 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:57.638402    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf" podUID="b2e9988f-011a-4a80-ad65-d54e5266e8dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:01:57.835478 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:57.835448    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9"]
Nov 02 02:01:57.836165 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:57.836151    5639 cpu_manager.go:414] "containerMap: removing container" podUID="babf6f72-f00a-45eb-8295-1054a1d61850" containerName="revoke-keys"
Nov 02 02:01:57.836227 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:57.836167    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="babf6f72-f00a-45eb-8295-1054a1d61850" containerName="revoke-keys"
Nov 02 02:01:57.836227 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:57.836188    5639 cpu_manager.go:414] "containerMap: removing container" podUID="25cd82fe-a02f-45cf-9a41-bdbaea642b30" containerName="istio-proxy"
Nov 02 02:01:57.836227 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:57.836193    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25cd82fe-a02f-45cf-9a41-bdbaea642b30" containerName="istio-proxy"
Nov 02 02:01:57.836227 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:57.836223    5639 cpu_manager.go:414] "containerMap: removing container" podUID="25cd82fe-a02f-45cf-9a41-bdbaea642b30" containerName="istio-proxy"
Nov 02 02:01:57.836376 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:57.836229    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25cd82fe-a02f-45cf-9a41-bdbaea642b30" containerName="istio-proxy"
Nov 02 02:01:57.836376 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:57.836347    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="25cd82fe-a02f-45cf-9a41-bdbaea642b30" containerName="istio-proxy"
Nov 02 02:01:57.836376 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:57.836361    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="babf6f72-f00a-45eb-8295-1054a1d61850" containerName="revoke-keys"
Nov 02 02:01:57.841346 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:57.841318    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9"
Nov 02 02:01:57.897797 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:57.897757    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9"]
Nov 02 02:01:57.942725 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:57.942685    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:01:57.942954 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:57.942922    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:01:57.948917 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:57.948894    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:01:57.949030 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:57.948948    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:01:58.005129 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:58.005098    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4hnn\" (UniqueName: \"kubernetes.io/projected/a0c26c59-2897-4252-b545-9da191ac7735-kube-api-access-x4hnn\") pod \"maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9\" (UID: \"a0c26c59-2897-4252-b545-9da191ac7735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9"
Nov 02 02:01:58.005306 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:58.005294    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c26c59-2897-4252-b545-9da191ac7735-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9\" (UID: \"a0c26c59-2897-4252-b545-9da191ac7735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9"
Nov 02 02:01:58.005422 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:58.005401    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c26c59-2897-4252-b545-9da191ac7735-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9\" (UID: \"a0c26c59-2897-4252-b545-9da191ac7735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9"
Nov 02 02:01:58.107165 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:58.107074    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4hnn\" (UniqueName: \"kubernetes.io/projected/a0c26c59-2897-4252-b545-9da191ac7735-kube-api-access-x4hnn\") pod \"maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9\" (UID: \"a0c26c59-2897-4252-b545-9da191ac7735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9"
Nov 02 02:01:58.107379 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:58.107249    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c26c59-2897-4252-b545-9da191ac7735-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9\" (UID: \"a0c26c59-2897-4252-b545-9da191ac7735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9"
Nov 02 02:01:58.107379 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:58.107365    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c26c59-2897-4252-b545-9da191ac7735-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9\" (UID: \"a0c26c59-2897-4252-b545-9da191ac7735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9"
Nov 02 02:01:58.107848 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:58.107824    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c26c59-2897-4252-b545-9da191ac7735-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9\" (UID: \"a0c26c59-2897-4252-b545-9da191ac7735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9"
Nov 02 02:01:58.109881 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:58.109863    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c26c59-2897-4252-b545-9da191ac7735-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9\" (UID: \"a0c26c59-2897-4252-b545-9da191ac7735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9"
Nov 02 02:01:58.168991 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:58.168949    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4hnn\" (UniqueName: \"kubernetes.io/projected/a0c26c59-2897-4252-b545-9da191ac7735-kube-api-access-x4hnn\") pod \"maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9\" (UID: \"a0c26c59-2897-4252-b545-9da191ac7735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9"
Nov 02 02:01:58.455432 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:58.455348    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9"
Nov 02 02:01:58.879838 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:58.879811    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9"]
Nov 02 02:01:58.880720 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:01:58.880695    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c26c59_2897_4252_b545_9da191ac7735.slice/crio-5ab708d187cb68c43a1b48cd36c38ac1f79b42002e2532ef74d7ef925beabae2 WatchSource:0}: Error finding container 5ab708d187cb68c43a1b48cd36c38ac1f79b42002e2532ef74d7ef925beabae2: Status 404 returned error can't find the container with id 5ab708d187cb68c43a1b48cd36c38ac1f79b42002e2532ef74d7ef925beabae2
Nov 02 02:01:59.084128 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.084094    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9" event={"ID":"a0c26c59-2897-4252-b545-9da191ac7735","Type":"ContainerStarted","Data":"5ab708d187cb68c43a1b48cd36c38ac1f79b42002e2532ef74d7ef925beabae2"}
Nov 02 02:01:59.506260 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.506228    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 02:01:59.634274 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.634243    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-data\") pod \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") "
Nov 02 02:01:59.634473 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.634313    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-token\" (UniqueName: \"kubernetes.io/projected/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-token\") pod \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") "
Nov 02 02:01:59.634473 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.634375    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istiod-ca-cert\") pod \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") "
Nov 02 02:01:59.634473 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.634392    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-workload-certs\") pod \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") "
Nov 02 02:01:59.634473 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.634446    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-credential-socket\") pod \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") "
Nov 02 02:01:59.634685 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.634539    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-workload-socket\") pod \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") "
Nov 02 02:01:59.634685 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.634590    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e9988f-011a-4a80-ad65-d54e5266e8dd-kube-api-access-k2mcw\" (UniqueName: \"kubernetes.io/projected/b2e9988f-011a-4a80-ad65-d54e5266e8dd-kube-api-access-k2mcw\") pod \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") "
Nov 02 02:01:59.634804 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.634689    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-envoy\") pod \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") "
Nov 02 02:01:59.634868 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.634583    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-data" pod "b2e9988f-011a-4a80-ad65-d54e5266e8dd" (UID: "b2e9988f-011a-4a80-ad65-d54e5266e8dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:59.634868 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.634839    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-workload-certs" pod "b2e9988f-011a-4a80-ad65-d54e5266e8dd" (UID: "b2e9988f-011a-4a80-ad65-d54e5266e8dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:59.634868 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.634854    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-workload-socket" pod "b2e9988f-011a-4a80-ad65-d54e5266e8dd" (UID: "b2e9988f-011a-4a80-ad65-d54e5266e8dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:59.634868 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.634825    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-credential-socket" pod "b2e9988f-011a-4a80-ad65-d54e5266e8dd" (UID: "b2e9988f-011a-4a80-ad65-d54e5266e8dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:59.635075 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.634861    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istiod-ca-cert" pod "b2e9988f-011a-4a80-ad65-d54e5266e8dd" (UID: "b2e9988f-011a-4a80-ad65-d54e5266e8dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:59.635075 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.634931    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-podinfo\") pod \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\" (UID: \"b2e9988f-011a-4a80-ad65-d54e5266e8dd\") "
Nov 02 02:01:59.635812 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.635785    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:59.635950 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.635815    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:59.635950 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.635830    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:59.635950 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.635844    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:59.635950 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.635858    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:59.636836 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.636807    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-token" pod "b2e9988f-011a-4a80-ad65-d54e5266e8dd" (UID: "b2e9988f-011a-4a80-ad65-d54e5266e8dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:59.636957 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.636890    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e9988f-011a-4a80-ad65-d54e5266e8dd-kube-api-access-k2mcw" pod "b2e9988f-011a-4a80-ad65-d54e5266e8dd" (UID: "b2e9988f-011a-4a80-ad65-d54e5266e8dd"). InnerVolumeSpecName "kube-api-access-k2mcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:59.637005 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.636994    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-envoy" pod "b2e9988f-011a-4a80-ad65-d54e5266e8dd" (UID: "b2e9988f-011a-4a80-ad65-d54e5266e8dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:59.637307 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.637253    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-podinfo" pod "b2e9988f-011a-4a80-ad65-d54e5266e8dd" (UID: "b2e9988f-011a-4a80-ad65-d54e5266e8dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:01:59.737237 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.737158    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:59.737237 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.737184    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:59.737237 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.737195    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2mcw\" (UniqueName: \"kubernetes.io/projected/b2e9988f-011a-4a80-ad65-d54e5266e8dd-kube-api-access-k2mcw\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:59.737237 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:01:59.737204    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e9988f-011a-4a80-ad65-d54e5266e8dd-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:00.094180 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:00.094144    5639 generic.go:356] "Generic (PLEG): container finished" podID="b2e9988f-011a-4a80-ad65-d54e5266e8dd" containerID="397a4da7f345c8732a14e16debefca5d88a4f39c3cbc66991ce02c624224df3d" exitCode=0
Nov 02 02:02:00.094412 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:00.094202    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"
Nov 02 02:02:00.094412 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:00.094229    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf" event={"ID":"b2e9988f-011a-4a80-ad65-d54e5266e8dd","Type":"ContainerDied","Data":"397a4da7f345c8732a14e16debefca5d88a4f39c3cbc66991ce02c624224df3d"}
Nov 02 02:02:00.094412 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:00.094296    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf" event={"ID":"b2e9988f-011a-4a80-ad65-d54e5266e8dd","Type":"ContainerDied","Data":"adb52972c3ed260414b20cb8b5f7df8e04e023b2d1cc9ec96dac2d2346bcfc43"}
Nov 02 02:02:00.094412 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:00.094327    5639 scope.go:122] "RemoveContainer" containerID="397a4da7f345c8732a14e16debefca5d88a4f39c3cbc66991ce02c624224df3d"
Nov 02 02:02:00.096176 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:00.096151    5639 generic.go:356] "Generic (PLEG): container finished" podID="a0c26c59-2897-4252-b545-9da191ac7735" containerID="303e529c7e536e658c21bfab6cbc318624484d964f891c37af7d3f76805d8fe6" exitCode=0
Nov 02 02:02:00.096305 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:00.096233    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9" event={"ID":"a0c26c59-2897-4252-b545-9da191ac7735","Type":"ContainerDied","Data":"303e529c7e536e658c21bfab6cbc318624484d964f891c37af7d3f76805d8fe6"}
Nov 02 02:02:00.106627 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:00.106608    5639 scope.go:122] "RemoveContainer" containerID="ed62dd24cfa8cc54208107256e01590ea5f2a6782a2a17b58ee356598ee85378"
Nov 02 02:02:00.128406 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:00.128381    5639 scope.go:122] "RemoveContainer" containerID="397a4da7f345c8732a14e16debefca5d88a4f39c3cbc66991ce02c624224df3d"
Nov 02 02:02:00.128758 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:02:00.128738    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"397a4da7f345c8732a14e16debefca5d88a4f39c3cbc66991ce02c624224df3d\": container with ID starting with 397a4da7f345c8732a14e16debefca5d88a4f39c3cbc66991ce02c624224df3d not found: ID does not exist" containerID="397a4da7f345c8732a14e16debefca5d88a4f39c3cbc66991ce02c624224df3d"
Nov 02 02:02:00.128810 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:00.128768    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"397a4da7f345c8732a14e16debefca5d88a4f39c3cbc66991ce02c624224df3d"} err="failed to get container status \"397a4da7f345c8732a14e16debefca5d88a4f39c3cbc66991ce02c624224df3d\": rpc error: code = NotFound desc = could not find container \"397a4da7f345c8732a14e16debefca5d88a4f39c3cbc66991ce02c624224df3d\": container with ID starting with 397a4da7f345c8732a14e16debefca5d88a4f39c3cbc66991ce02c624224df3d not found: ID does not exist"
Nov 02 02:02:00.128810 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:00.128791    5639 scope.go:122] "RemoveContainer" containerID="ed62dd24cfa8cc54208107256e01590ea5f2a6782a2a17b58ee356598ee85378"
Nov 02 02:02:00.129093 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:02:00.129078    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed62dd24cfa8cc54208107256e01590ea5f2a6782a2a17b58ee356598ee85378\": container with ID starting with ed62dd24cfa8cc54208107256e01590ea5f2a6782a2a17b58ee356598ee85378 not found: ID does not exist" containerID="ed62dd24cfa8cc54208107256e01590ea5f2a6782a2a17b58ee356598ee85378"
Nov 02 02:02:00.129136 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:00.129097    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed62dd24cfa8cc54208107256e01590ea5f2a6782a2a17b58ee356598ee85378"} err="failed to get container status \"ed62dd24cfa8cc54208107256e01590ea5f2a6782a2a17b58ee356598ee85378\": rpc error: code = NotFound desc = could not find container \"ed62dd24cfa8cc54208107256e01590ea5f2a6782a2a17b58ee356598ee85378\": container with ID starting with ed62dd24cfa8cc54208107256e01590ea5f2a6782a2a17b58ee356598ee85378 not found: ID does not exist"
Nov 02 02:02:00.261078 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:00.261037    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"]
Nov 02 02:02:00.310616 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:00.310586    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-109b07-openshift-default-54786bf495-brglf"]
Nov 02 02:02:01.165564 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:01.165429    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"]
Nov 02 02:02:01.224348 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:01.224313    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2e9988f-011a-4a80-ad65-d54e5266e8dd" path="/var/lib/kubelet/pods/b2e9988f-011a-4a80-ad65-d54e5266e8dd/volumes"
Nov 02 02:02:01.298870 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:01.298845    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9"
Nov 02 02:02:01.377777 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:01.377699    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0c26c59-2897-4252-b545-9da191ac7735-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c26c59-2897-4252-b545-9da191ac7735-maas-api-xks-ca\") pod \"a0c26c59-2897-4252-b545-9da191ac7735\" (UID: \"a0c26c59-2897-4252-b545-9da191ac7735\") "
Nov 02 02:02:01.377777 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:01.377744    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c26c59-2897-4252-b545-9da191ac7735-kube-api-access-x4hnn\" (UniqueName: \"kubernetes.io/projected/a0c26c59-2897-4252-b545-9da191ac7735-kube-api-access-x4hnn\") pod \"a0c26c59-2897-4252-b545-9da191ac7735\" (UID: \"a0c26c59-2897-4252-b545-9da191ac7735\") "
Nov 02 02:02:01.377965 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:01.377900    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0c26c59-2897-4252-b545-9da191ac7735-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c26c59-2897-4252-b545-9da191ac7735-maas-api-openshift-ca\") pod \"a0c26c59-2897-4252-b545-9da191ac7735\" (UID: \"a0c26c59-2897-4252-b545-9da191ac7735\") "
Nov 02 02:02:01.378314 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:01.378242    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c26c59-2897-4252-b545-9da191ac7735-maas-api-openshift-ca" pod "a0c26c59-2897-4252-b545-9da191ac7735" (UID: "a0c26c59-2897-4252-b545-9da191ac7735"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:01.378613 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:01.378532    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c26c59-2897-4252-b545-9da191ac7735-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:01.380028 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:01.380000    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c26c59-2897-4252-b545-9da191ac7735-maas-api-xks-ca" pod "a0c26c59-2897-4252-b545-9da191ac7735" (UID: "a0c26c59-2897-4252-b545-9da191ac7735"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:02:01.380228 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:01.380212    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c26c59-2897-4252-b545-9da191ac7735-kube-api-access-x4hnn" pod "a0c26c59-2897-4252-b545-9da191ac7735" (UID: "a0c26c59-2897-4252-b545-9da191ac7735"). InnerVolumeSpecName "kube-api-access-x4hnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:01.479983 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:01.479944    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c26c59-2897-4252-b545-9da191ac7735-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:01.479983 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:01.479982    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4hnn\" (UniqueName: \"kubernetes.io/projected/a0c26c59-2897-4252-b545-9da191ac7735-kube-api-access-x4hnn\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:02.114054 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:02.114020    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9"
Nov 02 02:02:02.114054 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:02.114047    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9" event={"ID":"a0c26c59-2897-4252-b545-9da191ac7735","Type":"ContainerDied","Data":"5ab708d187cb68c43a1b48cd36c38ac1f79b42002e2532ef74d7ef925beabae2"}
Nov 02 02:02:02.114294 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:02.114095    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ab708d187cb68c43a1b48cd36c38ac1f79b42002e2532ef74d7ef925beabae2"
Nov 02 02:02:02.986708 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:02.986668    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"]
Nov 02 02:02:02.987505 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:02.986907    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s" podUID="a02ec44c-7c77-41b5-967e-b83c696ee76f" containerName="istio-proxy" containerID="cri-o://930f1aeb207c71c12a4e2dc9df3ef4778ecf24aa9012bdd793df0002e6f3e848" gracePeriod=30
Nov 02 02:02:03.036872 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:03.036834    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"]
Nov 02 02:02:03.037551 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:03.037489    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8" podUID="56a5cefb-fa4f-49bf-9213-3951fa917f74" containerName="istio-proxy" containerID="cri-o://9382717286a1010c5cac676f38cd0c8008061f8fe4c1e3cfd43ce110dbfc8e00" gracePeriod=30
Nov 02 02:02:03.040982 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:03.040954    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"]
Nov 02 02:02:03.041343 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:03.041311    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt" podUID="565a8a37-83e6-45ef-9b22-d524cbd6231e" containerName="istio-proxy" containerID="cri-o://240557fdef98eb9cefd0ef6c19e0c51291c576be8b58f3d94fe3eb4c15d60150" gracePeriod=30
Nov 02 02:02:08.183395 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.178264    5639 generic.go:356] "Generic (PLEG): container finished" podID="a02ec44c-7c77-41b5-967e-b83c696ee76f" containerID="930f1aeb207c71c12a4e2dc9df3ef4778ecf24aa9012bdd793df0002e6f3e848" exitCode=0
Nov 02 02:02:08.183395 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.178346    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s" event={"ID":"a02ec44c-7c77-41b5-967e-b83c696ee76f","Type":"ContainerDied","Data":"930f1aeb207c71c12a4e2dc9df3ef4778ecf24aa9012bdd793df0002e6f3e848"}
Nov 02 02:02:08.183395 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.180405    5639 generic.go:356] "Generic (PLEG): container finished" podID="56a5cefb-fa4f-49bf-9213-3951fa917f74" containerID="9382717286a1010c5cac676f38cd0c8008061f8fe4c1e3cfd43ce110dbfc8e00" exitCode=0
Nov 02 02:02:08.183395 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.180488    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8" event={"ID":"56a5cefb-fa4f-49bf-9213-3951fa917f74","Type":"ContainerDied","Data":"9382717286a1010c5cac676f38cd0c8008061f8fe4c1e3cfd43ce110dbfc8e00"}
Nov 02 02:02:08.183395 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.182980    5639 generic.go:356] "Generic (PLEG): container finished" podID="565a8a37-83e6-45ef-9b22-d524cbd6231e" containerID="240557fdef98eb9cefd0ef6c19e0c51291c576be8b58f3d94fe3eb4c15d60150" exitCode=0
Nov 02 02:02:08.183395 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.183127    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt" event={"ID":"565a8a37-83e6-45ef-9b22-d524cbd6231e","Type":"ContainerDied","Data":"240557fdef98eb9cefd0ef6c19e0c51291c576be8b58f3d94fe3eb4c15d60150"}
Nov 02 02:02:08.296223 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.296194    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 02:02:08.366596 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.366568    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 02:02:08.391209 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.391184    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 02:02:08.408295 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.408257    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-podinfo\") pod \"565a8a37-83e6-45ef-9b22-d524cbd6231e\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") "
Nov 02 02:02:08.408441 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.408354    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-data\") pod \"565a8a37-83e6-45ef-9b22-d524cbd6231e\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") "
Nov 02 02:02:08.408441 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.408376    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/565a8a37-83e6-45ef-9b22-d524cbd6231e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565a8a37-83e6-45ef-9b22-d524cbd6231e-istiod-ca-cert\") pod \"565a8a37-83e6-45ef-9b22-d524cbd6231e\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") "
Nov 02 02:02:08.408441 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.408407    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a02ec44c-7c77-41b5-967e-b83c696ee76f-kube-api-access-92qg5\" (UniqueName: \"kubernetes.io/projected/a02ec44c-7c77-41b5-967e-b83c696ee76f-kube-api-access-92qg5\") pod \"a02ec44c-7c77-41b5-967e-b83c696ee76f\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") "
Nov 02 02:02:08.408565 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.408472    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-credential-socket\") pod \"565a8a37-83e6-45ef-9b22-d524cbd6231e\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") "
Nov 02 02:02:08.408565 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.408505    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-data\") pod \"a02ec44c-7c77-41b5-967e-b83c696ee76f\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") "
Nov 02 02:02:08.408674 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.408564    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-credential-socket\") pod \"a02ec44c-7c77-41b5-967e-b83c696ee76f\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") "
Nov 02 02:02:08.408674 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.408594    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/565a8a37-83e6-45ef-9b22-d524cbd6231e-kube-api-access-pr4vb\" (UniqueName: \"kubernetes.io/projected/565a8a37-83e6-45ef-9b22-d524cbd6231e-kube-api-access-pr4vb\") pod \"565a8a37-83e6-45ef-9b22-d524cbd6231e\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") "
Nov 02 02:02:08.408674 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.408625    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-workload-certs\") pod \"565a8a37-83e6-45ef-9b22-d524cbd6231e\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") "
Nov 02 02:02:08.408841 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.408679    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a02ec44c-7c77-41b5-967e-b83c696ee76f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a02ec44c-7c77-41b5-967e-b83c696ee76f-istiod-ca-cert\") pod \"a02ec44c-7c77-41b5-967e-b83c696ee76f\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") "
Nov 02 02:02:08.408841 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.408739    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-token\" (UniqueName: \"kubernetes.io/projected/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-token\") pod \"a02ec44c-7c77-41b5-967e-b83c696ee76f\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") "
Nov 02 02:02:08.408841 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.408834    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-token\" (UniqueName: \"kubernetes.io/projected/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-token\") pod \"565a8a37-83e6-45ef-9b22-d524cbd6231e\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") "
Nov 02 02:02:08.409010 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.408675    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-data" pod "565a8a37-83e6-45ef-9b22-d524cbd6231e" (UID: "565a8a37-83e6-45ef-9b22-d524cbd6231e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:08.409010 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.408850    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/565a8a37-83e6-45ef-9b22-d524cbd6231e-istiod-ca-cert" pod "565a8a37-83e6-45ef-9b22-d524cbd6231e" (UID: "565a8a37-83e6-45ef-9b22-d524cbd6231e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:08.409010 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.408854    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-data" pod "a02ec44c-7c77-41b5-967e-b83c696ee76f" (UID: "a02ec44c-7c77-41b5-967e-b83c696ee76f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:08.409010 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.408892    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-workload-certs\") pod \"a02ec44c-7c77-41b5-967e-b83c696ee76f\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") "
Nov 02 02:02:08.409010 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.408944    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-workload-socket\") pod \"a02ec44c-7c77-41b5-967e-b83c696ee76f\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") "
Nov 02 02:02:08.409299 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.409021    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-envoy\") pod \"a02ec44c-7c77-41b5-967e-b83c696ee76f\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") "
Nov 02 02:02:08.409299 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.409066    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-podinfo\") pod \"a02ec44c-7c77-41b5-967e-b83c696ee76f\" (UID: \"a02ec44c-7c77-41b5-967e-b83c696ee76f\") "
Nov 02 02:02:08.409299 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.409084    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-envoy\") pod \"565a8a37-83e6-45ef-9b22-d524cbd6231e\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") "
Nov 02 02:02:08.409299 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.409149    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-workload-socket\") pod \"565a8a37-83e6-45ef-9b22-d524cbd6231e\" (UID: \"565a8a37-83e6-45ef-9b22-d524cbd6231e\") "
Nov 02 02:02:08.409299 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.409184    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-credential-socket" pod "a02ec44c-7c77-41b5-967e-b83c696ee76f" (UID: "a02ec44c-7c77-41b5-967e-b83c696ee76f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:08.409299 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.409218    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-credential-socket" pod "565a8a37-83e6-45ef-9b22-d524cbd6231e" (UID: "565a8a37-83e6-45ef-9b22-d524cbd6231e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:08.409633 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.409236    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-workload-certs" pod "565a8a37-83e6-45ef-9b22-d524cbd6231e" (UID: "565a8a37-83e6-45ef-9b22-d524cbd6231e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:08.409633 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.409319    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-workload-certs" pod "a02ec44c-7c77-41b5-967e-b83c696ee76f" (UID: "a02ec44c-7c77-41b5-967e-b83c696ee76f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:08.409633 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.409451    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a02ec44c-7c77-41b5-967e-b83c696ee76f-istiod-ca-cert" pod "a02ec44c-7c77-41b5-967e-b83c696ee76f" (UID: "a02ec44c-7c77-41b5-967e-b83c696ee76f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:08.409633 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.409547    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-workload-socket" pod "a02ec44c-7c77-41b5-967e-b83c696ee76f" (UID: "a02ec44c-7c77-41b5-967e-b83c696ee76f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:08.410020 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.409740    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-workload-socket" pod "565a8a37-83e6-45ef-9b22-d524cbd6231e" (UID: "565a8a37-83e6-45ef-9b22-d524cbd6231e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:08.410020 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.409907    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.410020 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.409920    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a02ec44c-7c77-41b5-967e-b83c696ee76f-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.410020 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.409929    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.410020 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.409937    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.410020 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.409947    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.410020 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.409955    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.410020 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.409962    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565a8a37-83e6-45ef-9b22-d524cbd6231e-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.410020 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.409973    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.410020 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.409985    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.410020 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.409993    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.411544 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.411514    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-podinfo" pod "565a8a37-83e6-45ef-9b22-d524cbd6231e" (UID: "565a8a37-83e6-45ef-9b22-d524cbd6231e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:02:08.411802 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.411763    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-token" pod "a02ec44c-7c77-41b5-967e-b83c696ee76f" (UID: "a02ec44c-7c77-41b5-967e-b83c696ee76f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:08.412051 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.412024    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-envoy" pod "a02ec44c-7c77-41b5-967e-b83c696ee76f" (UID: "a02ec44c-7c77-41b5-967e-b83c696ee76f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:08.412265 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.412236    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565a8a37-83e6-45ef-9b22-d524cbd6231e-kube-api-access-pr4vb" pod "565a8a37-83e6-45ef-9b22-d524cbd6231e" (UID: "565a8a37-83e6-45ef-9b22-d524cbd6231e"). InnerVolumeSpecName "kube-api-access-pr4vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:08.412387 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.412312    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-envoy" pod "565a8a37-83e6-45ef-9b22-d524cbd6231e" (UID: "565a8a37-83e6-45ef-9b22-d524cbd6231e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:08.412387 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.412318    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a02ec44c-7c77-41b5-967e-b83c696ee76f-kube-api-access-92qg5" pod "a02ec44c-7c77-41b5-967e-b83c696ee76f" (UID: "a02ec44c-7c77-41b5-967e-b83c696ee76f"). InnerVolumeSpecName "kube-api-access-92qg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:08.412608 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.412590    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-token" pod "565a8a37-83e6-45ef-9b22-d524cbd6231e" (UID: "565a8a37-83e6-45ef-9b22-d524cbd6231e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:08.413208 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.413190    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-podinfo" pod "a02ec44c-7c77-41b5-967e-b83c696ee76f" (UID: "a02ec44c-7c77-41b5-967e-b83c696ee76f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:02:08.511517 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.511473    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56a5cefb-fa4f-49bf-9213-3951fa917f74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56a5cefb-fa4f-49bf-9213-3951fa917f74-istiod-ca-cert\") pod \"56a5cefb-fa4f-49bf-9213-3951fa917f74\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") "
Nov 02 02:02:08.511725 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.511580    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-credential-socket\") pod \"56a5cefb-fa4f-49bf-9213-3951fa917f74\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") "
Nov 02 02:02:08.511725 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.511631    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-workload-certs\") pod \"56a5cefb-fa4f-49bf-9213-3951fa917f74\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") "
Nov 02 02:02:08.511725 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.511665    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-workload-socket\") pod \"56a5cefb-fa4f-49bf-9213-3951fa917f74\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") "
Nov 02 02:02:08.511725 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.511701    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-data\") pod \"56a5cefb-fa4f-49bf-9213-3951fa917f74\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") "
Nov 02 02:02:08.511941 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.511742    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a5cefb-fa4f-49bf-9213-3951fa917f74-kube-api-access-4sxt2\" (UniqueName: \"kubernetes.io/projected/56a5cefb-fa4f-49bf-9213-3951fa917f74-kube-api-access-4sxt2\") pod \"56a5cefb-fa4f-49bf-9213-3951fa917f74\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") "
Nov 02 02:02:08.511941 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.511827    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-envoy\") pod \"56a5cefb-fa4f-49bf-9213-3951fa917f74\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") "
Nov 02 02:02:08.511941 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.511910    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-credential-socket" pod "56a5cefb-fa4f-49bf-9213-3951fa917f74" (UID: "56a5cefb-fa4f-49bf-9213-3951fa917f74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:08.512090 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.511957    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a5cefb-fa4f-49bf-9213-3951fa917f74-istiod-ca-cert" pod "56a5cefb-fa4f-49bf-9213-3951fa917f74" (UID: "56a5cefb-fa4f-49bf-9213-3951fa917f74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:08.512090 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.512045    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-token\" (UniqueName: \"kubernetes.io/projected/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-token\") pod \"56a5cefb-fa4f-49bf-9213-3951fa917f74\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") "
Nov 02 02:02:08.512194 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.512092    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-data" pod "56a5cefb-fa4f-49bf-9213-3951fa917f74" (UID: "56a5cefb-fa4f-49bf-9213-3951fa917f74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:08.512194 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.512119    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-podinfo\") pod \"56a5cefb-fa4f-49bf-9213-3951fa917f74\" (UID: \"56a5cefb-fa4f-49bf-9213-3951fa917f74\") "
Nov 02 02:02:08.512739 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.512575    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-workload-socket" pod "56a5cefb-fa4f-49bf-9213-3951fa917f74" (UID: "56a5cefb-fa4f-49bf-9213-3951fa917f74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:08.512739 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.512688    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-workload-certs" pod "56a5cefb-fa4f-49bf-9213-3951fa917f74" (UID: "56a5cefb-fa4f-49bf-9213-3951fa917f74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:08.512925 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.512803    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56a5cefb-fa4f-49bf-9213-3951fa917f74-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.512925 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.512820    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.512925 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.512835    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.512925 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.512848    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.512925 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.512860    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.512925 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.512875    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.512925 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.512889    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.512925 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.512902    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.512925 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.512916    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a02ec44c-7c77-41b5-967e-b83c696ee76f-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.513673 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.512929    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.513673 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.512942    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565a8a37-83e6-45ef-9b22-d524cbd6231e-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.513673 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.512956    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92qg5\" (UniqueName: \"kubernetes.io/projected/a02ec44c-7c77-41b5-967e-b83c696ee76f-kube-api-access-92qg5\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.513673 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.512971    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr4vb\" (UniqueName: \"kubernetes.io/projected/565a8a37-83e6-45ef-9b22-d524cbd6231e-kube-api-access-pr4vb\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.514838 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.514813    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-envoy" pod "56a5cefb-fa4f-49bf-9213-3951fa917f74" (UID: "56a5cefb-fa4f-49bf-9213-3951fa917f74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:08.514940 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.514839    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a5cefb-fa4f-49bf-9213-3951fa917f74-kube-api-access-4sxt2" pod "56a5cefb-fa4f-49bf-9213-3951fa917f74" (UID: "56a5cefb-fa4f-49bf-9213-3951fa917f74"). InnerVolumeSpecName "kube-api-access-4sxt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:08.515050 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.515029    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-podinfo" pod "56a5cefb-fa4f-49bf-9213-3951fa917f74" (UID: "56a5cefb-fa4f-49bf-9213-3951fa917f74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:02:08.515378 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.515358    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-token" pod "56a5cefb-fa4f-49bf-9213-3951fa917f74" (UID: "56a5cefb-fa4f-49bf-9213-3951fa917f74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:08.614892 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.614860    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.614892 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.614889    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.614892 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.614901    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sxt2\" (UniqueName: \"kubernetes.io/projected/56a5cefb-fa4f-49bf-9213-3951fa917f74-kube-api-access-4sxt2\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.615122 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:08.614910    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56a5cefb-fa4f-49bf-9213-3951fa917f74-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:09.192866 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:09.192834    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"
Nov 02 02:02:09.193359 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:09.192859    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s" event={"ID":"a02ec44c-7c77-41b5-967e-b83c696ee76f","Type":"ContainerDied","Data":"a4aabef8c3aae08170f3ca6651629db144011efca8dd69ddf437c10218f0632e"}
Nov 02 02:02:09.193359 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:09.192948    5639 scope.go:122] "RemoveContainer" containerID="930f1aeb207c71c12a4e2dc9df3ef4778ecf24aa9012bdd793df0002e6f3e848"
Nov 02 02:02:09.194675 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:09.194654    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"
Nov 02 02:02:09.194822 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:09.194651    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8" event={"ID":"56a5cefb-fa4f-49bf-9213-3951fa917f74","Type":"ContainerDied","Data":"101934cccb93ab141b9456abb611bf07013443c303f8a4fc2d8095e1ede4a05b"}
Nov 02 02:02:09.196183 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:09.196161    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt" event={"ID":"565a8a37-83e6-45ef-9b22-d524cbd6231e","Type":"ContainerDied","Data":"8f861252c25cf7f4d7b62bda7da20167c8b885d72cd5ab535a8a4e226d194b8a"}
Nov 02 02:02:09.196183 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:09.196176    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"
Nov 02 02:02:09.204470 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:09.204454    5639 scope.go:122] "RemoveContainer" containerID="9382717286a1010c5cac676f38cd0c8008061f8fe4c1e3cfd43ce110dbfc8e00"
Nov 02 02:02:09.216078 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:09.216059    5639 scope.go:122] "RemoveContainer" containerID="240557fdef98eb9cefd0ef6c19e0c51291c576be8b58f3d94fe3eb4c15d60150"
Nov 02 02:02:09.310480 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:09.310395    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc"]
Nov 02 02:02:09.359079 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:09.359050    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"]
Nov 02 02:02:09.364993 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:09.364962    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kvb8"]
Nov 02 02:02:09.475845 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:09.475810    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"]
Nov 02 02:02:09.479786 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:09.479753    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-nqz7s"]
Nov 02 02:02:09.590249 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:09.590213    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"]
Nov 02 02:02:09.596315 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:09.596272    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b22tt"]
Nov 02 02:02:11.221609 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:11.221579    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="565a8a37-83e6-45ef-9b22-d524cbd6231e" path="/var/lib/kubelet/pods/565a8a37-83e6-45ef-9b22-d524cbd6231e/volumes"
Nov 02 02:02:11.222150 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:11.222134    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a5cefb-fa4f-49bf-9213-3951fa917f74" path="/var/lib/kubelet/pods/56a5cefb-fa4f-49bf-9213-3951fa917f74/volumes"
Nov 02 02:02:11.222675 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:11.222660    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a02ec44c-7c77-41b5-967e-b83c696ee76f" path="/var/lib/kubelet/pods/a02ec44c-7c77-41b5-967e-b83c696ee76f/volumes"
Nov 02 02:02:14.311252 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:14.311214    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc" podUID="2c6540a2-93f1-4867-ab73-bf3643ef90b5" containerName="payload-processing" containerID="cri-o://23a6469a6741ad85f389aaf7d656c10df0ca3169706e73c49d181c1e53276b76" gracePeriod=25
Nov 02 02:02:16.205533 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:16.205470    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main" containerID="cri-o://9e68f4c82721f73f52a8e70236c5f8b47e05b8d4d8e2f7ed394fa6ef869543ff" gracePeriod=45
Nov 02 02:02:16.475346 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:16.475263    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 02:02:16.634315 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:16.634244    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-dshm\") pod \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\" (UID: \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\") "
Nov 02 02:02:16.634528 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:16.634349    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-model-cache\") pod \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\" (UID: \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\") "
Nov 02 02:02:16.634528 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:16.634480    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/169d2fdb-6588-4d7e-9892-2d12c0e017b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/169d2fdb-6588-4d7e-9892-2d12c0e017b6-tls-certs\") pod \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\" (UID: \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\") "
Nov 02 02:02:16.634728 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:16.634607    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/169d2fdb-6588-4d7e-9892-2d12c0e017b6-kube-api-access-l87jl\" (UniqueName: \"kubernetes.io/projected/169d2fdb-6588-4d7e-9892-2d12c0e017b6-kube-api-access-l87jl\") pod \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\" (UID: \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\") "
Nov 02 02:02:16.634728 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:16.634640    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-model-cache" pod "169d2fdb-6588-4d7e-9892-2d12c0e017b6" (UID: "169d2fdb-6588-4d7e-9892-2d12c0e017b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:16.634728 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:16.634653    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-home\" (UniqueName: \"kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-home\") pod \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\" (UID: \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\") "
Nov 02 02:02:16.634897 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:16.634743    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-tmp-dir\") pod \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\" (UID: \"169d2fdb-6588-4d7e-9892-2d12c0e017b6\") "
Nov 02 02:02:16.634949 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:16.634899    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-home" pod "169d2fdb-6588-4d7e-9892-2d12c0e017b6" (UID: "169d2fdb-6588-4d7e-9892-2d12c0e017b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:16.636159 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:16.635461    5639 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-model-cache\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:16.636159 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:16.635487    5639 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-home\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:16.636159 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:16.635506    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-tmp-dir" pod "169d2fdb-6588-4d7e-9892-2d12c0e017b6" (UID: "169d2fdb-6588-4d7e-9892-2d12c0e017b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:16.637865 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:16.637836    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/169d2fdb-6588-4d7e-9892-2d12c0e017b6-tls-certs" pod "169d2fdb-6588-4d7e-9892-2d12c0e017b6" (UID: "169d2fdb-6588-4d7e-9892-2d12c0e017b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:02:16.637991 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:16.637884    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-dshm" pod "169d2fdb-6588-4d7e-9892-2d12c0e017b6" (UID: "169d2fdb-6588-4d7e-9892-2d12c0e017b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:16.637991 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:16.637901    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169d2fdb-6588-4d7e-9892-2d12c0e017b6-kube-api-access-l87jl" pod "169d2fdb-6588-4d7e-9892-2d12c0e017b6" (UID: "169d2fdb-6588-4d7e-9892-2d12c0e017b6"). InnerVolumeSpecName "kube-api-access-l87jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:16.736820 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:16.736736    5639 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/169d2fdb-6588-4d7e-9892-2d12c0e017b6-tls-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:16.736820 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:16.736767    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l87jl\" (UniqueName: \"kubernetes.io/projected/169d2fdb-6588-4d7e-9892-2d12c0e017b6-kube-api-access-l87jl\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:16.736820 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:16.736782    5639 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-tmp-dir\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:16.736820 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:16.736797    5639 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/169d2fdb-6588-4d7e-9892-2d12c0e017b6-dshm\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:17.257705 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:17.257668    5639 generic.go:356] "Generic (PLEG): container finished" podID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerID="9e68f4c82721f73f52a8e70236c5f8b47e05b8d4d8e2f7ed394fa6ef869543ff" exitCode=0
Nov 02 02:02:17.258135 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:17.257724    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"
Nov 02 02:02:17.258135 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:17.257752    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" event={"ID":"169d2fdb-6588-4d7e-9892-2d12c0e017b6","Type":"ContainerDied","Data":"9e68f4c82721f73f52a8e70236c5f8b47e05b8d4d8e2f7ed394fa6ef869543ff"}
Nov 02 02:02:17.258135 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:17.257798    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5" event={"ID":"169d2fdb-6588-4d7e-9892-2d12c0e017b6","Type":"ContainerDied","Data":"e3e4fac4aa265480c4e85b8806d366073b10965b6b277b0275105e9aa0ab39e9"}
Nov 02 02:02:17.258135 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:17.257817    5639 scope.go:122] "RemoveContainer" containerID="9e68f4c82721f73f52a8e70236c5f8b47e05b8d4d8e2f7ed394fa6ef869543ff"
Nov 02 02:02:17.268680 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:17.268659    5639 scope.go:122] "RemoveContainer" containerID="9e68f4c82721f73f52a8e70236c5f8b47e05b8d4d8e2f7ed394fa6ef869543ff"
Nov 02 02:02:17.268985 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:02:17.268952    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e68f4c82721f73f52a8e70236c5f8b47e05b8d4d8e2f7ed394fa6ef869543ff\": container with ID starting with 9e68f4c82721f73f52a8e70236c5f8b47e05b8d4d8e2f7ed394fa6ef869543ff not found: ID does not exist" containerID="9e68f4c82721f73f52a8e70236c5f8b47e05b8d4d8e2f7ed394fa6ef869543ff"
Nov 02 02:02:17.269120 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:17.268984    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e68f4c82721f73f52a8e70236c5f8b47e05b8d4d8e2f7ed394fa6ef869543ff"} err="failed to get container status \"9e68f4c82721f73f52a8e70236c5f8b47e05b8d4d8e2f7ed394fa6ef869543ff\": rpc error: code = NotFound desc = could not find container \"9e68f4c82721f73f52a8e70236c5f8b47e05b8d4d8e2f7ed394fa6ef869543ff\": container with ID starting with 9e68f4c82721f73f52a8e70236c5f8b47e05b8d4d8e2f7ed394fa6ef869543ff not found: ID does not exist"
Nov 02 02:02:17.423466 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:17.423435    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"]
Nov 02 02:02:17.473146 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:17.473107    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b71ba5/distinct-2-w0-b71ba5-kserve-864b7d9fb8-s59s5"]
Nov 02 02:02:19.220364 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:19.220333    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" path="/var/lib/kubelet/pods/169d2fdb-6588-4d7e-9892-2d12c0e017b6/volumes"
Nov 02 02:02:19.690455 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:19.690414    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc" podUID="2c6540a2-93f1-4867-ab73-bf3643ef90b5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:02:19.971184 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:19.971105    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r"]
Nov 02 02:02:20.020818 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:20.020788    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d700e9-ngk6r"]
Nov 02 02:02:21.220101 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.220069    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="babf6f72-f00a-45eb-8295-1054a1d61850" path="/var/lib/kubelet/pods/babf6f72-f00a-45eb-8295-1054a1d61850/volumes"
Nov 02 02:02:21.411943 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.411911    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-269235-7ns97"]
Nov 02 02:02:21.412622 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.412607    5639 cpu_manager.go:414] "containerMap: removing container" podUID="565a8a37-83e6-45ef-9b22-d524cbd6231e" containerName="istio-proxy"
Nov 02 02:02:21.412683 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.412624    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="565a8a37-83e6-45ef-9b22-d524cbd6231e" containerName="istio-proxy"
Nov 02 02:02:21.412683 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.412656    5639 cpu_manager.go:414] "containerMap: removing container" podUID="b2e9988f-011a-4a80-ad65-d54e5266e8dd" containerName="istio-proxy"
Nov 02 02:02:21.412683 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.412660    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e9988f-011a-4a80-ad65-d54e5266e8dd" containerName="istio-proxy"
Nov 02 02:02:21.412683 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.412673    5639 cpu_manager.go:414] "containerMap: removing container" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main"
Nov 02 02:02:21.412683 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.412678    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main"
Nov 02 02:02:21.412839 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.412697    5639 cpu_manager.go:414] "containerMap: removing container" podUID="56a5cefb-fa4f-49bf-9213-3951fa917f74" containerName="istio-proxy"
Nov 02 02:02:21.412839 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.412706    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a5cefb-fa4f-49bf-9213-3951fa917f74" containerName="istio-proxy"
Nov 02 02:02:21.412839 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.412719    5639 cpu_manager.go:414] "containerMap: removing container" podUID="a02ec44c-7c77-41b5-967e-b83c696ee76f" containerName="istio-proxy"
Nov 02 02:02:21.412839 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.412723    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a02ec44c-7c77-41b5-967e-b83c696ee76f" containerName="istio-proxy"
Nov 02 02:02:21.412839 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.412729    5639 cpu_manager.go:414] "containerMap: removing container" podUID="b2e9988f-011a-4a80-ad65-d54e5266e8dd" containerName="istio-proxy"
Nov 02 02:02:21.412839 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.412734    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e9988f-011a-4a80-ad65-d54e5266e8dd" containerName="istio-proxy"
Nov 02 02:02:21.412839 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.412740    5639 cpu_manager.go:414] "containerMap: removing container" podUID="a0c26c59-2897-4252-b545-9da191ac7735" containerName="revoke-keys"
Nov 02 02:02:21.412839 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.412745    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c26c59-2897-4252-b545-9da191ac7735" containerName="revoke-keys"
Nov 02 02:02:21.412839 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.412840    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e9988f-011a-4a80-ad65-d54e5266e8dd" containerName="istio-proxy"
Nov 02 02:02:21.413117 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.412848    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="169d2fdb-6588-4d7e-9892-2d12c0e017b6" containerName="main"
Nov 02 02:02:21.413117 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.412853    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="25cd82fe-a02f-45cf-9a41-bdbaea642b30" containerName="istio-proxy"
Nov 02 02:02:21.413117 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.412860    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a5cefb-fa4f-49bf-9213-3951fa917f74" containerName="istio-proxy"
Nov 02 02:02:21.413117 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.412865    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="a02ec44c-7c77-41b5-967e-b83c696ee76f" containerName="istio-proxy"
Nov 02 02:02:21.413117 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.412869    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e9988f-011a-4a80-ad65-d54e5266e8dd" containerName="istio-proxy"
Nov 02 02:02:21.413117 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.412875    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c26c59-2897-4252-b545-9da191ac7735" containerName="revoke-keys"
Nov 02 02:02:21.413117 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.412887    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="565a8a37-83e6-45ef-9b22-d524cbd6231e" containerName="istio-proxy"
Nov 02 02:02:21.418033 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.418012    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-269235-7ns97"
Nov 02 02:02:21.474151 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.474073    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-269235-7ns97"]
Nov 02 02:02:21.518903 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.518874    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:02:21.519113 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.518881    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:02:21.534799 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.534766    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd43750-3a7f-45cd-b4dc-57dec370a295-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-269235-7ns97\" (UID: \"3bd43750-3a7f-45cd-b4dc-57dec370a295\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-269235-7ns97"
Nov 02 02:02:21.534966 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.534885    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8zgd\" (UniqueName: \"kubernetes.io/projected/3bd43750-3a7f-45cd-b4dc-57dec370a295-kube-api-access-b8zgd\") pod \"maas-api-revoke-keys-e2e-api-a-269235-7ns97\" (UID: \"3bd43750-3a7f-45cd-b4dc-57dec370a295\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-269235-7ns97"
Nov 02 02:02:21.535052 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.535034    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd43750-3a7f-45cd-b4dc-57dec370a295-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-269235-7ns97\" (UID: \"3bd43750-3a7f-45cd-b4dc-57dec370a295\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-269235-7ns97"
Nov 02 02:02:21.556063 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.556027    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:02:21.556200 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.556143    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:02:21.636017 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.635983    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd43750-3a7f-45cd-b4dc-57dec370a295-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-269235-7ns97\" (UID: \"3bd43750-3a7f-45cd-b4dc-57dec370a295\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-269235-7ns97"
Nov 02 02:02:21.636216 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.636065    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8zgd\" (UniqueName: \"kubernetes.io/projected/3bd43750-3a7f-45cd-b4dc-57dec370a295-kube-api-access-b8zgd\") pod \"maas-api-revoke-keys-e2e-api-a-269235-7ns97\" (UID: \"3bd43750-3a7f-45cd-b4dc-57dec370a295\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-269235-7ns97"
Nov 02 02:02:21.636314 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.636227    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd43750-3a7f-45cd-b4dc-57dec370a295-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-269235-7ns97\" (UID: \"3bd43750-3a7f-45cd-b4dc-57dec370a295\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-269235-7ns97"
Nov 02 02:02:21.636734 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.636711    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd43750-3a7f-45cd-b4dc-57dec370a295-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-269235-7ns97\" (UID: \"3bd43750-3a7f-45cd-b4dc-57dec370a295\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-269235-7ns97"
Nov 02 02:02:21.638745 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.638726    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd43750-3a7f-45cd-b4dc-57dec370a295-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-269235-7ns97\" (UID: \"3bd43750-3a7f-45cd-b4dc-57dec370a295\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-269235-7ns97"
Nov 02 02:02:21.697658 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.697621    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8zgd\" (UniqueName: \"kubernetes.io/projected/3bd43750-3a7f-45cd-b4dc-57dec370a295-kube-api-access-b8zgd\") pod \"maas-api-revoke-keys-e2e-api-a-269235-7ns97\" (UID: \"3bd43750-3a7f-45cd-b4dc-57dec370a295\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-269235-7ns97"
Nov 02 02:02:21.732427 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:21.732349    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-269235-7ns97"
Nov 02 02:02:22.115842 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:22.115806    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-269235-7ns97"]
Nov 02 02:02:22.116215 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:02:22.116183    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd43750_3a7f_45cd_b4dc_57dec370a295.slice/crio-e955783546cd8e616f1ed632c95dcad0aa39017a8437bd3b2b419b4a13cec1d4 WatchSource:0}: Error finding container e955783546cd8e616f1ed632c95dcad0aa39017a8437bd3b2b419b4a13cec1d4: Status 404 returned error can't find the container with id e955783546cd8e616f1ed632c95dcad0aa39017a8437bd3b2b419b4a13cec1d4
Nov 02 02:02:22.329366 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:22.329327    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-269235-7ns97" event={"ID":"3bd43750-3a7f-45cd-b4dc-57dec370a295","Type":"ContainerStarted","Data":"e955783546cd8e616f1ed632c95dcad0aa39017a8437bd3b2b419b4a13cec1d4"}
Nov 02 02:02:23.340262 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:23.340225    5639 generic.go:356] "Generic (PLEG): container finished" podID="3bd43750-3a7f-45cd-b4dc-57dec370a295" containerID="4ae07a32499e951453baf1ba687644250b8354ae3787ef94136cbabc2d5666c1" exitCode=0
Nov 02 02:02:23.340710 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:23.340309    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-269235-7ns97" event={"ID":"3bd43750-3a7f-45cd-b4dc-57dec370a295","Type":"ContainerDied","Data":"4ae07a32499e951453baf1ba687644250b8354ae3787ef94136cbabc2d5666c1"}
Nov 02 02:02:24.505233 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:24.505199    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm"]
Nov 02 02:02:24.509994 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:24.509970    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm"
Nov 02 02:02:24.517297 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:24.517267    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-269235-7ns97"
Nov 02 02:02:24.567765 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:24.567734    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm"]
Nov 02 02:02:24.609372 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:24.609291    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bd43750-3a7f-45cd-b4dc-57dec370a295-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd43750-3a7f-45cd-b4dc-57dec370a295-maas-api-xks-ca\") pod \"3bd43750-3a7f-45cd-b4dc-57dec370a295\" (UID: \"3bd43750-3a7f-45cd-b4dc-57dec370a295\") "
Nov 02 02:02:24.609545 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:24.609451    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bd43750-3a7f-45cd-b4dc-57dec370a295-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd43750-3a7f-45cd-b4dc-57dec370a295-maas-api-openshift-ca\") pod \"3bd43750-3a7f-45cd-b4dc-57dec370a295\" (UID: \"3bd43750-3a7f-45cd-b4dc-57dec370a295\") "
Nov 02 02:02:24.609545 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:24.609491    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bd43750-3a7f-45cd-b4dc-57dec370a295-kube-api-access-b8zgd\" (UniqueName: \"kubernetes.io/projected/3bd43750-3a7f-45cd-b4dc-57dec370a295-kube-api-access-b8zgd\") pod \"3bd43750-3a7f-45cd-b4dc-57dec370a295\" (UID: \"3bd43750-3a7f-45cd-b4dc-57dec370a295\") "
Nov 02 02:02:24.610502 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:24.609810    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d3ccdcb-0da5-408d-a15b-a8be4770b4be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm\" (UID: \"0d3ccdcb-0da5-408d-a15b-a8be4770b4be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm"
Nov 02 02:02:24.610502 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:24.609851    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bd43750-3a7f-45cd-b4dc-57dec370a295-maas-api-openshift-ca" pod "3bd43750-3a7f-45cd-b4dc-57dec370a295" (UID: "3bd43750-3a7f-45cd-b4dc-57dec370a295"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:24.610502 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:24.609996    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d3ccdcb-0da5-408d-a15b-a8be4770b4be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm\" (UID: \"0d3ccdcb-0da5-408d-a15b-a8be4770b4be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm"
Nov 02 02:02:24.610502 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:24.610046    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5fxj\" (UniqueName: \"kubernetes.io/projected/0d3ccdcb-0da5-408d-a15b-a8be4770b4be-kube-api-access-g5fxj\") pod \"maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm\" (UID: \"0d3ccdcb-0da5-408d-a15b-a8be4770b4be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm"
Nov 02 02:02:24.610502 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:24.610335    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd43750-3a7f-45cd-b4dc-57dec370a295-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:24.611922 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:24.611899    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bd43750-3a7f-45cd-b4dc-57dec370a295-maas-api-xks-ca" pod "3bd43750-3a7f-45cd-b4dc-57dec370a295" (UID: "3bd43750-3a7f-45cd-b4dc-57dec370a295"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:02:24.612581 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:24.612563    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd43750-3a7f-45cd-b4dc-57dec370a295-kube-api-access-b8zgd" pod "3bd43750-3a7f-45cd-b4dc-57dec370a295" (UID: "3bd43750-3a7f-45cd-b4dc-57dec370a295"). InnerVolumeSpecName "kube-api-access-b8zgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:24.711414 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:24.711374    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d3ccdcb-0da5-408d-a15b-a8be4770b4be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm\" (UID: \"0d3ccdcb-0da5-408d-a15b-a8be4770b4be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm"
Nov 02 02:02:24.711580 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:24.711513    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d3ccdcb-0da5-408d-a15b-a8be4770b4be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm\" (UID: \"0d3ccdcb-0da5-408d-a15b-a8be4770b4be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm"
Nov 02 02:02:24.711580 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:24.711557    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5fxj\" (UniqueName: \"kubernetes.io/projected/0d3ccdcb-0da5-408d-a15b-a8be4770b4be-kube-api-access-g5fxj\") pod \"maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm\" (UID: \"0d3ccdcb-0da5-408d-a15b-a8be4770b4be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm"
Nov 02 02:02:24.711664 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:24.711653    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8zgd\" (UniqueName: \"kubernetes.io/projected/3bd43750-3a7f-45cd-b4dc-57dec370a295-kube-api-access-b8zgd\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:24.711703 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:24.711673    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd43750-3a7f-45cd-b4dc-57dec370a295-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:24.712054 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:24.712033    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d3ccdcb-0da5-408d-a15b-a8be4770b4be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm\" (UID: \"0d3ccdcb-0da5-408d-a15b-a8be4770b4be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm"
Nov 02 02:02:24.713982 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:24.713963    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d3ccdcb-0da5-408d-a15b-a8be4770b4be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm\" (UID: \"0d3ccdcb-0da5-408d-a15b-a8be4770b4be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm"
Nov 02 02:02:24.772987 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:24.772952    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5fxj\" (UniqueName: \"kubernetes.io/projected/0d3ccdcb-0da5-408d-a15b-a8be4770b4be-kube-api-access-g5fxj\") pod \"maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm\" (UID: \"0d3ccdcb-0da5-408d-a15b-a8be4770b4be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm"
Nov 02 02:02:24.826340 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:24.826309    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm"
Nov 02 02:02:25.243832 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:25.243797    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm"]
Nov 02 02:02:25.245405 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:02:25.245380    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3ccdcb_0da5_408d_a15b_a8be4770b4be.slice/crio-1a228b90470073bfdf05ff81c792f70c95b3605eb0387586ef4a2df5dbeb1e12 WatchSource:0}: Error finding container 1a228b90470073bfdf05ff81c792f70c95b3605eb0387586ef4a2df5dbeb1e12: Status 404 returned error can't find the container with id 1a228b90470073bfdf05ff81c792f70c95b3605eb0387586ef4a2df5dbeb1e12
Nov 02 02:02:25.355706 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:25.355678    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-269235-7ns97"
Nov 02 02:02:25.355893 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:25.355676    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-269235-7ns97" event={"ID":"3bd43750-3a7f-45cd-b4dc-57dec370a295","Type":"ContainerDied","Data":"e955783546cd8e616f1ed632c95dcad0aa39017a8437bd3b2b419b4a13cec1d4"}
Nov 02 02:02:25.355893 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:25.355801    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e955783546cd8e616f1ed632c95dcad0aa39017a8437bd3b2b419b4a13cec1d4"
Nov 02 02:02:25.357107 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:25.357082    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm" event={"ID":"0d3ccdcb-0da5-408d-a15b-a8be4770b4be","Type":"ContainerStarted","Data":"1a228b90470073bfdf05ff81c792f70c95b3605eb0387586ef4a2df5dbeb1e12"}
Nov 02 02:02:26.368771 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:26.368730    5639 generic.go:356] "Generic (PLEG): container finished" podID="0d3ccdcb-0da5-408d-a15b-a8be4770b4be" containerID="fa560a3e3809883d470202ce02868185ebf1606d92626d5f39aa82eb781fdbbd" exitCode=0
Nov 02 02:02:26.369251 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:26.368823    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm" event={"ID":"0d3ccdcb-0da5-408d-a15b-a8be4770b4be","Type":"ContainerDied","Data":"fa560a3e3809883d470202ce02868185ebf1606d92626d5f39aa82eb781fdbbd"}
Nov 02 02:02:27.425908 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:02:27.425845    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:02:27.516058 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:27.516026    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"]
Nov 02 02:02:27.516365 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:27.516298    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8" podUID="ceaab0ba-9b59-4a1a-839d-bbd4322f9629" containerName="istio-proxy" containerID="cri-o://baa3b1d8fb5a9a269982b4667258075b61ed6b57be50bb68153cf3c7f402f5a7" gracePeriod=30
Nov 02 02:02:27.639362 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:27.634502    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm"
Nov 02 02:02:27.754443 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:27.754410    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d3ccdcb-0da5-408d-a15b-a8be4770b4be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d3ccdcb-0da5-408d-a15b-a8be4770b4be-maas-api-openshift-ca\") pod \"0d3ccdcb-0da5-408d-a15b-a8be4770b4be\" (UID: \"0d3ccdcb-0da5-408d-a15b-a8be4770b4be\") "
Nov 02 02:02:27.754633 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:27.754506    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d3ccdcb-0da5-408d-a15b-a8be4770b4be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d3ccdcb-0da5-408d-a15b-a8be4770b4be-maas-api-xks-ca\") pod \"0d3ccdcb-0da5-408d-a15b-a8be4770b4be\" (UID: \"0d3ccdcb-0da5-408d-a15b-a8be4770b4be\") "
Nov 02 02:02:27.754633 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:27.754598    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d3ccdcb-0da5-408d-a15b-a8be4770b4be-kube-api-access-g5fxj\" (UniqueName: \"kubernetes.io/projected/0d3ccdcb-0da5-408d-a15b-a8be4770b4be-kube-api-access-g5fxj\") pod \"0d3ccdcb-0da5-408d-a15b-a8be4770b4be\" (UID: \"0d3ccdcb-0da5-408d-a15b-a8be4770b4be\") "
Nov 02 02:02:27.754828 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:27.754792    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d3ccdcb-0da5-408d-a15b-a8be4770b4be-maas-api-openshift-ca" pod "0d3ccdcb-0da5-408d-a15b-a8be4770b4be" (UID: "0d3ccdcb-0da5-408d-a15b-a8be4770b4be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:27.755257 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:27.755238    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d3ccdcb-0da5-408d-a15b-a8be4770b4be-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:27.756718 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:27.756693    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d3ccdcb-0da5-408d-a15b-a8be4770b4be-maas-api-xks-ca" pod "0d3ccdcb-0da5-408d-a15b-a8be4770b4be" (UID: "0d3ccdcb-0da5-408d-a15b-a8be4770b4be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:02:27.756804 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:27.756769    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3ccdcb-0da5-408d-a15b-a8be4770b4be-kube-api-access-g5fxj" pod "0d3ccdcb-0da5-408d-a15b-a8be4770b4be" (UID: "0d3ccdcb-0da5-408d-a15b-a8be4770b4be"). InnerVolumeSpecName "kube-api-access-g5fxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:27.856986 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:27.856952    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d3ccdcb-0da5-408d-a15b-a8be4770b4be-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:27.856986 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:27.856983    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5fxj\" (UniqueName: \"kubernetes.io/projected/0d3ccdcb-0da5-408d-a15b-a8be4770b4be-kube-api-access-g5fxj\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:28.328395 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:28.328353    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm"]
Nov 02 02:02:28.329384 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:28.329361    5639 cpu_manager.go:414] "containerMap: removing container" podUID="0d3ccdcb-0da5-408d-a15b-a8be4770b4be" containerName="revoke-keys"
Nov 02 02:02:28.329447 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:28.329388    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d3ccdcb-0da5-408d-a15b-a8be4770b4be" containerName="revoke-keys"
Nov 02 02:02:28.329447 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:28.329410    5639 cpu_manager.go:414] "containerMap: removing container" podUID="3bd43750-3a7f-45cd-b4dc-57dec370a295" containerName="revoke-keys"
Nov 02 02:02:28.329447 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:28.329418    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bd43750-3a7f-45cd-b4dc-57dec370a295" containerName="revoke-keys"
Nov 02 02:02:28.329694 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:28.329680    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d3ccdcb-0da5-408d-a15b-a8be4770b4be" containerName="revoke-keys"
Nov 02 02:02:28.329733 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:28.329701    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bd43750-3a7f-45cd-b4dc-57dec370a295" containerName="revoke-keys"
Nov 02 02:02:28.333740 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:28.333716    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm"
Nov 02 02:02:28.390779 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:28.390747    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm"]
Nov 02 02:02:28.392313 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:28.392260    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm"
Nov 02 02:02:28.392437 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:28.392336    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm" event={"ID":"0d3ccdcb-0da5-408d-a15b-a8be4770b4be","Type":"ContainerDied","Data":"1a228b90470073bfdf05ff81c792f70c95b3605eb0387586ef4a2df5dbeb1e12"}
Nov 02 02:02:28.392437 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:28.392389    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a228b90470073bfdf05ff81c792f70c95b3605eb0387586ef4a2df5dbeb1e12"
Nov 02 02:02:28.463249 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:28.463219    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98zjg\" (UniqueName: \"kubernetes.io/projected/4a51de5a-8641-47b0-be51-21d1be052318-kube-api-access-98zjg\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm\" (UID: \"4a51de5a-8641-47b0-be51-21d1be052318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm"
Nov 02 02:02:28.463707 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:28.463268    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a51de5a-8641-47b0-be51-21d1be052318-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm\" (UID: \"4a51de5a-8641-47b0-be51-21d1be052318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm"
Nov 02 02:02:28.463707 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:28.463481    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a51de5a-8641-47b0-be51-21d1be052318-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm\" (UID: \"4a51de5a-8641-47b0-be51-21d1be052318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm"
Nov 02 02:02:28.565156 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:28.565117    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98zjg\" (UniqueName: \"kubernetes.io/projected/4a51de5a-8641-47b0-be51-21d1be052318-kube-api-access-98zjg\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm\" (UID: \"4a51de5a-8641-47b0-be51-21d1be052318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm"
Nov 02 02:02:28.565156 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:28.565161    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a51de5a-8641-47b0-be51-21d1be052318-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm\" (UID: \"4a51de5a-8641-47b0-be51-21d1be052318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm"
Nov 02 02:02:28.565443 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:28.565259    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a51de5a-8641-47b0-be51-21d1be052318-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm\" (UID: \"4a51de5a-8641-47b0-be51-21d1be052318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm"
Nov 02 02:02:28.565937 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:28.565910    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a51de5a-8641-47b0-be51-21d1be052318-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm\" (UID: \"4a51de5a-8641-47b0-be51-21d1be052318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm"
Nov 02 02:02:28.567666 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:28.567642    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a51de5a-8641-47b0-be51-21d1be052318-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm\" (UID: \"4a51de5a-8641-47b0-be51-21d1be052318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm"
Nov 02 02:02:28.626043 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:28.625964    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98zjg\" (UniqueName: \"kubernetes.io/projected/4a51de5a-8641-47b0-be51-21d1be052318-kube-api-access-98zjg\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm\" (UID: \"4a51de5a-8641-47b0-be51-21d1be052318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm"
Nov 02 02:02:28.649061 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:28.649032    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm"
Nov 02 02:02:29.001335 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:29.001240    5639 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:02:29.001335 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:29.001308    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8" podUID="ceaab0ba-9b59-4a1a-839d-bbd4322f9629" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:02:29.027779 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:29.027750    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm"]
Nov 02 02:02:29.029555 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:02:29.029529    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a51de5a_8641_47b0_be51_21d1be052318.slice/crio-92f731ac1128c8dcfac1591c6e75c1acfd7e35dc0487f7def8cc4cd3543cd9c5 WatchSource:0}: Error finding container 92f731ac1128c8dcfac1591c6e75c1acfd7e35dc0487f7def8cc4cd3543cd9c5: Status 404 returned error can't find the container with id 92f731ac1128c8dcfac1591c6e75c1acfd7e35dc0487f7def8cc4cd3543cd9c5
Nov 02 02:02:29.439970 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:29.439934    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm" event={"ID":"4a51de5a-8641-47b0-be51-21d1be052318","Type":"ContainerStarted","Data":"92f731ac1128c8dcfac1591c6e75c1acfd7e35dc0487f7def8cc4cd3543cd9c5"}
Nov 02 02:02:29.691581 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:29.691544    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc" podUID="2c6540a2-93f1-4867-ab73-bf3643ef90b5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:02:30.449506 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:30.449415    5639 generic.go:356] "Generic (PLEG): container finished" podID="4a51de5a-8641-47b0-be51-21d1be052318" containerID="2a0d8daaa97baca2065eaa3a704391d2149e685e823670cf017e1e1796380898" exitCode=0
Nov 02 02:02:30.449642 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:30.449503    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm" event={"ID":"4a51de5a-8641-47b0-be51-21d1be052318","Type":"ContainerDied","Data":"2a0d8daaa97baca2065eaa3a704391d2149e685e823670cf017e1e1796380898"}
Nov 02 02:02:31.564957 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:31.564917    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd"]
Nov 02 02:02:31.580028 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:31.579998    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-218802-hstmd"]
Nov 02 02:02:31.632097 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:31.632072    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm"
Nov 02 02:02:31.646435 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:31.646401    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a51de5a-8641-47b0-be51-21d1be052318-kube-api-access-98zjg\" (UniqueName: \"kubernetes.io/projected/4a51de5a-8641-47b0-be51-21d1be052318-kube-api-access-98zjg\") pod \"4a51de5a-8641-47b0-be51-21d1be052318\" (UID: \"4a51de5a-8641-47b0-be51-21d1be052318\") "
Nov 02 02:02:31.646579 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:31.646527    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a51de5a-8641-47b0-be51-21d1be052318-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a51de5a-8641-47b0-be51-21d1be052318-maas-api-xks-ca\") pod \"4a51de5a-8641-47b0-be51-21d1be052318\" (UID: \"4a51de5a-8641-47b0-be51-21d1be052318\") "
Nov 02 02:02:31.646699 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:31.646688    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a51de5a-8641-47b0-be51-21d1be052318-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a51de5a-8641-47b0-be51-21d1be052318-maas-api-openshift-ca\") pod \"4a51de5a-8641-47b0-be51-21d1be052318\" (UID: \"4a51de5a-8641-47b0-be51-21d1be052318\") "
Nov 02 02:02:31.647031 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:31.647005    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a51de5a-8641-47b0-be51-21d1be052318-maas-api-openshift-ca" pod "4a51de5a-8641-47b0-be51-21d1be052318" (UID: "4a51de5a-8641-47b0-be51-21d1be052318"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:31.648663 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:31.648640    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a51de5a-8641-47b0-be51-21d1be052318-maas-api-xks-ca" pod "4a51de5a-8641-47b0-be51-21d1be052318" (UID: "4a51de5a-8641-47b0-be51-21d1be052318"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:02:31.648744 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:31.648684    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a51de5a-8641-47b0-be51-21d1be052318-kube-api-access-98zjg" pod "4a51de5a-8641-47b0-be51-21d1be052318" (UID: "4a51de5a-8641-47b0-be51-21d1be052318"). InnerVolumeSpecName "kube-api-access-98zjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:31.748347 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:31.748190    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a51de5a-8641-47b0-be51-21d1be052318-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:31.748347 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:31.748233    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98zjg\" (UniqueName: \"kubernetes.io/projected/4a51de5a-8641-47b0-be51-21d1be052318-kube-api-access-98zjg\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:31.748347 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:31.748249    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a51de5a-8641-47b0-be51-21d1be052318-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:32.465896 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.465865    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm"
Nov 02 02:02:32.466085 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.465904    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm" event={"ID":"4a51de5a-8641-47b0-be51-21d1be052318","Type":"ContainerDied","Data":"92f731ac1128c8dcfac1591c6e75c1acfd7e35dc0487f7def8cc4cd3543cd9c5"}
Nov 02 02:02:32.466085 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.465950    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92f731ac1128c8dcfac1591c6e75c1acfd7e35dc0487f7def8cc4cd3543cd9c5"
Nov 02 02:02:32.804811 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.804785    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 02:02:32.862566 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.862536    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-workload-certs\") pod \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") "
Nov 02 02:02:32.862728 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.862612    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-workload-socket\") pod \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") "
Nov 02 02:02:32.862728 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.862653    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-envoy\") pod \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") "
Nov 02 02:02:32.862728 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.862706    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istiod-ca-cert\") pod \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") "
Nov 02 02:02:32.862728 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.862728    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-token\" (UniqueName: \"kubernetes.io/projected/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-token\") pod \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") "
Nov 02 02:02:32.862998 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.862791    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-workload-certs" pod "ceaab0ba-9b59-4a1a-839d-bbd4322f9629" (UID: "ceaab0ba-9b59-4a1a-839d-bbd4322f9629"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:32.862998 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.862849    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-podinfo\") pod \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") "
Nov 02 02:02:32.862998 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.862930    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-credential-socket\") pod \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") "
Nov 02 02:02:32.862998 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.862963    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-kube-api-access-z2snh\" (UniqueName: \"kubernetes.io/projected/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-kube-api-access-z2snh\") pod \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") "
Nov 02 02:02:32.862998 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.862967    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-workload-socket" pod "ceaab0ba-9b59-4a1a-839d-bbd4322f9629" (UID: "ceaab0ba-9b59-4a1a-839d-bbd4322f9629"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:32.863497 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.863114    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-data\") pod \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\" (UID: \"ceaab0ba-9b59-4a1a-839d-bbd4322f9629\") "
Nov 02 02:02:32.863497 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.863141    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-credential-socket" pod "ceaab0ba-9b59-4a1a-839d-bbd4322f9629" (UID: "ceaab0ba-9b59-4a1a-839d-bbd4322f9629"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:32.863497 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.863379    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-data" pod "ceaab0ba-9b59-4a1a-839d-bbd4322f9629" (UID: "ceaab0ba-9b59-4a1a-839d-bbd4322f9629"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:32.863660 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.863507    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istiod-ca-cert" pod "ceaab0ba-9b59-4a1a-839d-bbd4322f9629" (UID: "ceaab0ba-9b59-4a1a-839d-bbd4322f9629"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:32.863922 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.863829    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:32.863922 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.863852    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:32.863922 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.863868    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:32.863922 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.863883    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:32.863922 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.863896    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:32.865227 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.865204    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-token" pod "ceaab0ba-9b59-4a1a-839d-bbd4322f9629" (UID: "ceaab0ba-9b59-4a1a-839d-bbd4322f9629"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:32.865479 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.865454    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-kube-api-access-z2snh" pod "ceaab0ba-9b59-4a1a-839d-bbd4322f9629" (UID: "ceaab0ba-9b59-4a1a-839d-bbd4322f9629"). InnerVolumeSpecName "kube-api-access-z2snh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:32.865668 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.865650    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-envoy" pod "ceaab0ba-9b59-4a1a-839d-bbd4322f9629" (UID: "ceaab0ba-9b59-4a1a-839d-bbd4322f9629"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:32.865738 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.865662    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-podinfo" pod "ceaab0ba-9b59-4a1a-839d-bbd4322f9629" (UID: "ceaab0ba-9b59-4a1a-839d-bbd4322f9629"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:02:32.965844 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.965807    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:32.965844 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.965847    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:32.966046 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.965867    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:32.966046 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:32.965881    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2snh\" (UniqueName: \"kubernetes.io/projected/ceaab0ba-9b59-4a1a-839d-bbd4322f9629-kube-api-access-z2snh\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:33.221313 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:33.221261    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bccefd52-b090-41c8-88eb-83d79277cd74" path="/var/lib/kubelet/pods/bccefd52-b090-41c8-88eb-83d79277cd74/volumes"
Nov 02 02:02:33.476117 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:33.476031    5639 generic.go:356] "Generic (PLEG): container finished" podID="ceaab0ba-9b59-4a1a-839d-bbd4322f9629" containerID="baa3b1d8fb5a9a269982b4667258075b61ed6b57be50bb68153cf3c7f402f5a7" exitCode=0
Nov 02 02:02:33.476117 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:33.476083    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"
Nov 02 02:02:33.476117 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:33.476101    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8" event={"ID":"ceaab0ba-9b59-4a1a-839d-bbd4322f9629","Type":"ContainerDied","Data":"baa3b1d8fb5a9a269982b4667258075b61ed6b57be50bb68153cf3c7f402f5a7"}
Nov 02 02:02:33.476423 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:33.476142    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8" event={"ID":"ceaab0ba-9b59-4a1a-839d-bbd4322f9629","Type":"ContainerDied","Data":"9cb2374f3fe0bf66c64a0445ab1472f3ecdfe73471c3f6681fff1cec1923d423"}
Nov 02 02:02:33.476423 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:33.476164    5639 scope.go:122] "RemoveContainer" containerID="baa3b1d8fb5a9a269982b4667258075b61ed6b57be50bb68153cf3c7f402f5a7"
Nov 02 02:02:33.486908 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:33.486889    5639 scope.go:122] "RemoveContainer" containerID="aff79c7033179a98e70ec2051073a3ee7ad68c5c722173160fa4124cb1f1135b"
Nov 02 02:02:33.497243 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:33.497221    5639 scope.go:122] "RemoveContainer" containerID="baa3b1d8fb5a9a269982b4667258075b61ed6b57be50bb68153cf3c7f402f5a7"
Nov 02 02:02:33.497605 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:02:33.497582    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baa3b1d8fb5a9a269982b4667258075b61ed6b57be50bb68153cf3c7f402f5a7\": container with ID starting with baa3b1d8fb5a9a269982b4667258075b61ed6b57be50bb68153cf3c7f402f5a7 not found: ID does not exist" containerID="baa3b1d8fb5a9a269982b4667258075b61ed6b57be50bb68153cf3c7f402f5a7"
Nov 02 02:02:33.497667 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:33.497612    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa3b1d8fb5a9a269982b4667258075b61ed6b57be50bb68153cf3c7f402f5a7"} err="failed to get container status \"baa3b1d8fb5a9a269982b4667258075b61ed6b57be50bb68153cf3c7f402f5a7\": rpc error: code = NotFound desc = could not find container \"baa3b1d8fb5a9a269982b4667258075b61ed6b57be50bb68153cf3c7f402f5a7\": container with ID starting with baa3b1d8fb5a9a269982b4667258075b61ed6b57be50bb68153cf3c7f402f5a7 not found: ID does not exist"
Nov 02 02:02:33.497667 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:33.497631    5639 scope.go:122] "RemoveContainer" containerID="aff79c7033179a98e70ec2051073a3ee7ad68c5c722173160fa4124cb1f1135b"
Nov 02 02:02:33.497892 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:02:33.497867    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aff79c7033179a98e70ec2051073a3ee7ad68c5c722173160fa4124cb1f1135b\": container with ID starting with aff79c7033179a98e70ec2051073a3ee7ad68c5c722173160fa4124cb1f1135b not found: ID does not exist" containerID="aff79c7033179a98e70ec2051073a3ee7ad68c5c722173160fa4124cb1f1135b"
Nov 02 02:02:33.497972 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:33.497903    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff79c7033179a98e70ec2051073a3ee7ad68c5c722173160fa4124cb1f1135b"} err="failed to get container status \"aff79c7033179a98e70ec2051073a3ee7ad68c5c722173160fa4124cb1f1135b\": rpc error: code = NotFound desc = could not find container \"aff79c7033179a98e70ec2051073a3ee7ad68c5c722173160fa4124cb1f1135b\": container with ID starting with aff79c7033179a98e70ec2051073a3ee7ad68c5c722173160fa4124cb1f1135b not found: ID does not exist"
Nov 02 02:02:33.641356 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:33.641314    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"]
Nov 02 02:02:33.693815 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:33.693775    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d700e9-openshift-default-9f6cb7fcc-vrgz8"]
Nov 02 02:02:34.488010 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:34.487977    5639 generic.go:356] "Generic (PLEG): container finished" podID="2c6540a2-93f1-4867-ab73-bf3643ef90b5" containerID="23a6469a6741ad85f389aaf7d656c10df0ca3169706e73c49d181c1e53276b76" exitCode=0
Nov 02 02:02:34.488485 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:34.488025    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc" event={"ID":"2c6540a2-93f1-4867-ab73-bf3643ef90b5","Type":"ContainerDied","Data":"23a6469a6741ad85f389aaf7d656c10df0ca3169706e73c49d181c1e53276b76"}
Nov 02 02:02:34.587214 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:34.587184    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc"
Nov 02 02:02:34.687382 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:34.687348    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c6540a2-93f1-4867-ab73-bf3643ef90b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c6540a2-93f1-4867-ab73-bf3643ef90b5-plugins-config-volume\") pod \"2c6540a2-93f1-4867-ab73-bf3643ef90b5\" (UID: \"2c6540a2-93f1-4867-ab73-bf3643ef90b5\") "
Nov 02 02:02:34.687559 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:34.687437    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c6540a2-93f1-4867-ab73-bf3643ef90b5-kube-api-access-8hh6m\" (UniqueName: \"kubernetes.io/projected/2c6540a2-93f1-4867-ab73-bf3643ef90b5-kube-api-access-8hh6m\") pod \"2c6540a2-93f1-4867-ab73-bf3643ef90b5\" (UID: \"2c6540a2-93f1-4867-ab73-bf3643ef90b5\") "
Nov 02 02:02:34.687707 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:34.687685    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c6540a2-93f1-4867-ab73-bf3643ef90b5-plugins-config-volume" pod "2c6540a2-93f1-4867-ab73-bf3643ef90b5" (UID: "2c6540a2-93f1-4867-ab73-bf3643ef90b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:34.688010 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:34.687994    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c6540a2-93f1-4867-ab73-bf3643ef90b5-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:34.689723 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:34.689700    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c6540a2-93f1-4867-ab73-bf3643ef90b5-kube-api-access-8hh6m" pod "2c6540a2-93f1-4867-ab73-bf3643ef90b5" (UID: "2c6540a2-93f1-4867-ab73-bf3643ef90b5"). InnerVolumeSpecName "kube-api-access-8hh6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:34.789364 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:34.789325    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hh6m\" (UniqueName: \"kubernetes.io/projected/2c6540a2-93f1-4867-ab73-bf3643ef90b5-kube-api-access-8hh6m\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:35.219946 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:35.219909    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceaab0ba-9b59-4a1a-839d-bbd4322f9629" path="/var/lib/kubelet/pods/ceaab0ba-9b59-4a1a-839d-bbd4322f9629/volumes"
Nov 02 02:02:35.496359 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:35.496272    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc"
Nov 02 02:02:35.496359 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:35.496321    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc" event={"ID":"2c6540a2-93f1-4867-ab73-bf3643ef90b5","Type":"ContainerDied","Data":"87105ffac8a797b13ee975e96841492d98444c377ee3e41ed8ff68cc00b45a78"}
Nov 02 02:02:35.496856 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:35.496386    5639 scope.go:122] "RemoveContainer" containerID="23a6469a6741ad85f389aaf7d656c10df0ca3169706e73c49d181c1e53276b76"
Nov 02 02:02:35.661443 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:35.661410    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc"]
Nov 02 02:02:35.710967 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:35.710936    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b71ba5-55b4568655-42sdc"]
Nov 02 02:02:37.270318 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:37.270245    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c6540a2-93f1-4867-ab73-bf3643ef90b5" path="/var/lib/kubelet/pods/2c6540a2-93f1-4867-ab73-bf3643ef90b5/volumes"
Nov 02 02:02:38.099700 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:38.099662    5639 scope.go:122] "RemoveContainer" containerID="9f6fb0ca2b1ee9134dc34d5355bdcb81ea057d81ea41fa774ef54fff9fe06bbd"
Nov 02 02:02:39.406837 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:39.406806    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"]
Nov 02 02:02:39.407256 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:39.407022    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs" podUID="7542d266-7f8c-474a-a8dc-9848376e3993" containerName="istio-proxy" containerID="cri-o://0601eb1677436e92301b075d8cebd6c527170d03ffaa0895b231b2962dac6305" gracePeriod=30
Nov 02 02:02:40.258208 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.258170    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz"]
Nov 02 02:02:40.258837 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.258822    5639 cpu_manager.go:414] "containerMap: removing container" podUID="ceaab0ba-9b59-4a1a-839d-bbd4322f9629" containerName="istio-proxy"
Nov 02 02:02:40.258894 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.258839    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceaab0ba-9b59-4a1a-839d-bbd4322f9629" containerName="istio-proxy"
Nov 02 02:02:40.258894 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.258877    5639 cpu_manager.go:414] "containerMap: removing container" podUID="ceaab0ba-9b59-4a1a-839d-bbd4322f9629" containerName="istio-proxy"
Nov 02 02:02:40.258894 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.258882    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceaab0ba-9b59-4a1a-839d-bbd4322f9629" containerName="istio-proxy"
Nov 02 02:02:40.258894 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.258891    5639 cpu_manager.go:414] "containerMap: removing container" podUID="4a51de5a-8641-47b0-be51-21d1be052318" containerName="revoke-keys"
Nov 02 02:02:40.259024 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.258896    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a51de5a-8641-47b0-be51-21d1be052318" containerName="revoke-keys"
Nov 02 02:02:40.259024 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.258910    5639 cpu_manager.go:414] "containerMap: removing container" podUID="2c6540a2-93f1-4867-ab73-bf3643ef90b5" containerName="payload-processing"
Nov 02 02:02:40.259024 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.258915    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c6540a2-93f1-4867-ab73-bf3643ef90b5" containerName="payload-processing"
Nov 02 02:02:40.259024 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.258998    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c6540a2-93f1-4867-ab73-bf3643ef90b5" containerName="payload-processing"
Nov 02 02:02:40.259024 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.259008    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceaab0ba-9b59-4a1a-839d-bbd4322f9629" containerName="istio-proxy"
Nov 02 02:02:40.259024 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.259022    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceaab0ba-9b59-4a1a-839d-bbd4322f9629" containerName="istio-proxy"
Nov 02 02:02:40.259208 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.259027    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a51de5a-8641-47b0-be51-21d1be052318" containerName="revoke-keys"
Nov 02 02:02:40.264617 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.264592    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz"
Nov 02 02:02:40.323933 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.323903    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz"]
Nov 02 02:02:40.363033 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.362991    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62180ece-d5f2-48a4-8b9f-7cfe4e814121-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz\" (UID: \"62180ece-d5f2-48a4-8b9f-7cfe4e814121\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz"
Nov 02 02:02:40.363204 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.363056    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62180ece-d5f2-48a4-8b9f-7cfe4e814121-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz\" (UID: \"62180ece-d5f2-48a4-8b9f-7cfe4e814121\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz"
Nov 02 02:02:40.363204 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.363163    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkxqz\" (UniqueName: \"kubernetes.io/projected/62180ece-d5f2-48a4-8b9f-7cfe4e814121-kube-api-access-nkxqz\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz\" (UID: \"62180ece-d5f2-48a4-8b9f-7cfe4e814121\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz"
Nov 02 02:02:40.368021 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.368001    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:02:40.368150 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.368130    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:02:40.368210 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.368143    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:02:40.373500 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.373482    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:02:40.464262 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.464231    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkxqz\" (UniqueName: \"kubernetes.io/projected/62180ece-d5f2-48a4-8b9f-7cfe4e814121-kube-api-access-nkxqz\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz\" (UID: \"62180ece-d5f2-48a4-8b9f-7cfe4e814121\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz"
Nov 02 02:02:40.464761 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.464647    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62180ece-d5f2-48a4-8b9f-7cfe4e814121-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz\" (UID: \"62180ece-d5f2-48a4-8b9f-7cfe4e814121\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz"
Nov 02 02:02:40.464761 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.464708    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62180ece-d5f2-48a4-8b9f-7cfe4e814121-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz\" (UID: \"62180ece-d5f2-48a4-8b9f-7cfe4e814121\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz"
Nov 02 02:02:40.465406 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.465383    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62180ece-d5f2-48a4-8b9f-7cfe4e814121-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz\" (UID: \"62180ece-d5f2-48a4-8b9f-7cfe4e814121\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz"
Nov 02 02:02:40.467298 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.467263    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62180ece-d5f2-48a4-8b9f-7cfe4e814121-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz\" (UID: \"62180ece-d5f2-48a4-8b9f-7cfe4e814121\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz"
Nov 02 02:02:40.525023 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.524960    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkxqz\" (UniqueName: \"kubernetes.io/projected/62180ece-d5f2-48a4-8b9f-7cfe4e814121-kube-api-access-nkxqz\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz\" (UID: \"62180ece-d5f2-48a4-8b9f-7cfe4e814121\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz"
Nov 02 02:02:40.581572 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.581525    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz"
Nov 02 02:02:40.964398 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:40.964363    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz"]
Nov 02 02:02:40.964906 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:02:40.964874    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62180ece_d5f2_48a4_8b9f_7cfe4e814121.slice/crio-98e8ac54f1395e94e9d40cd61c6587f0e44a2574b4671ddf0e25f050732a3c9f WatchSource:0}: Error finding container 98e8ac54f1395e94e9d40cd61c6587f0e44a2574b4671ddf0e25f050732a3c9f: Status 404 returned error can't find the container with id 98e8ac54f1395e94e9d40cd61c6587f0e44a2574b4671ddf0e25f050732a3c9f
Nov 02 02:02:41.583414 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:41.583378    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz" event={"ID":"62180ece-d5f2-48a4-8b9f-7cfe4e814121","Type":"ContainerStarted","Data":"e14ba01c82cc1a65227cfb6dc775e592f39e17f9873c3d54e3b90e0c84b516f2"}
Nov 02 02:02:41.583911 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:41.583432    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz" event={"ID":"62180ece-d5f2-48a4-8b9f-7cfe4e814121","Type":"ContainerStarted","Data":"98e8ac54f1395e94e9d40cd61c6587f0e44a2574b4671ddf0e25f050732a3c9f"}
Nov 02 02:02:41.701508 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:41.701412    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz" podStartSLOduration=1.701398421 podStartE2EDuration="1.701398421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:02:41.699210997 +0000 UTC m=+2226.602893111" watchObservedRunningTime="2026-11-02 02:02:41.701398421 +0000 UTC m=+2226.605080534"
Nov 02 02:02:41.708546 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:41.708514    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92"]
Nov 02 02:02:41.773408 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:41.773372    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4"]
Nov 02 02:02:42.596011 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:42.595970    5639 generic.go:356] "Generic (PLEG): container finished" podID="62180ece-d5f2-48a4-8b9f-7cfe4e814121" containerID="e14ba01c82cc1a65227cfb6dc775e592f39e17f9873c3d54e3b90e0c84b516f2" exitCode=0
Nov 02 02:02:42.596456 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:42.596058    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz" event={"ID":"62180ece-d5f2-48a4-8b9f-7cfe4e814121","Type":"ContainerDied","Data":"e14ba01c82cc1a65227cfb6dc775e592f39e17f9873c3d54e3b90e0c84b516f2"}
Nov 02 02:02:43.779258 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:43.779223    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz"
Nov 02 02:02:43.862751 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:43.862670    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62180ece-d5f2-48a4-8b9f-7cfe4e814121-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62180ece-d5f2-48a4-8b9f-7cfe4e814121-maas-api-xks-ca\") pod \"62180ece-d5f2-48a4-8b9f-7cfe4e814121\" (UID: \"62180ece-d5f2-48a4-8b9f-7cfe4e814121\") "
Nov 02 02:02:43.862751 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:43.862721    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62180ece-d5f2-48a4-8b9f-7cfe4e814121-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62180ece-d5f2-48a4-8b9f-7cfe4e814121-maas-api-openshift-ca\") pod \"62180ece-d5f2-48a4-8b9f-7cfe4e814121\" (UID: \"62180ece-d5f2-48a4-8b9f-7cfe4e814121\") "
Nov 02 02:02:43.862989 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:43.862877    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62180ece-d5f2-48a4-8b9f-7cfe4e814121-kube-api-access-nkxqz\" (UniqueName: \"kubernetes.io/projected/62180ece-d5f2-48a4-8b9f-7cfe4e814121-kube-api-access-nkxqz\") pod \"62180ece-d5f2-48a4-8b9f-7cfe4e814121\" (UID: \"62180ece-d5f2-48a4-8b9f-7cfe4e814121\") "
Nov 02 02:02:43.863152 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:43.863119    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62180ece-d5f2-48a4-8b9f-7cfe4e814121-maas-api-openshift-ca" pod "62180ece-d5f2-48a4-8b9f-7cfe4e814121" (UID: "62180ece-d5f2-48a4-8b9f-7cfe4e814121"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:43.863485 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:43.863468    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62180ece-d5f2-48a4-8b9f-7cfe4e814121-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:43.865018 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:43.864997    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62180ece-d5f2-48a4-8b9f-7cfe4e814121-maas-api-xks-ca" pod "62180ece-d5f2-48a4-8b9f-7cfe4e814121" (UID: "62180ece-d5f2-48a4-8b9f-7cfe4e814121"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:02:43.865141 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:43.865125    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62180ece-d5f2-48a4-8b9f-7cfe4e814121-kube-api-access-nkxqz" pod "62180ece-d5f2-48a4-8b9f-7cfe4e814121" (UID: "62180ece-d5f2-48a4-8b9f-7cfe4e814121"). InnerVolumeSpecName "kube-api-access-nkxqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:43.964250 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:43.964220    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkxqz\" (UniqueName: \"kubernetes.io/projected/62180ece-d5f2-48a4-8b9f-7cfe4e814121-kube-api-access-nkxqz\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:43.964250 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:43.964249    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62180ece-d5f2-48a4-8b9f-7cfe4e814121-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:44.626014 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.625580    5639 generic.go:356] "Generic (PLEG): container finished" podID="7542d266-7f8c-474a-a8dc-9848376e3993" containerID="0601eb1677436e92301b075d8cebd6c527170d03ffaa0895b231b2962dac6305" exitCode=0
Nov 02 02:02:44.626014 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.625744    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs" event={"ID":"7542d266-7f8c-474a-a8dc-9848376e3993","Type":"ContainerDied","Data":"0601eb1677436e92301b075d8cebd6c527170d03ffaa0895b231b2962dac6305"}
Nov 02 02:02:44.626014 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.625843    5639 scope.go:122] "RemoveContainer" containerID="e662c189e08dcec93fdbe7cc8b5a01617477d671284cbf448c38a4df9f102569"
Nov 02 02:02:44.629708 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.629648    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz" event={"ID":"62180ece-d5f2-48a4-8b9f-7cfe4e814121","Type":"ContainerDied","Data":"98e8ac54f1395e94e9d40cd61c6587f0e44a2574b4671ddf0e25f050732a3c9f"}
Nov 02 02:02:44.632305 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.629908    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98e8ac54f1395e94e9d40cd61c6587f0e44a2574b4671ddf0e25f050732a3c9f"
Nov 02 02:02:44.632305 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.630113    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz"
Nov 02 02:02:44.664510 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.664486    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 02:02:44.775229 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.775203    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-istio-data\") pod \"7542d266-7f8c-474a-a8dc-9848376e3993\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") "
Nov 02 02:02:44.775399 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.775347    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-workload-certs\") pod \"7542d266-7f8c-474a-a8dc-9848376e3993\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") "
Nov 02 02:02:44.775399 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.775377    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-credential-socket\") pod \"7542d266-7f8c-474a-a8dc-9848376e3993\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") "
Nov 02 02:02:44.775526 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.775476    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-workload-socket\") pod \"7542d266-7f8c-474a-a8dc-9848376e3993\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") "
Nov 02 02:02:44.775526 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.775491    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-istio-data" pod "7542d266-7f8c-474a-a8dc-9848376e3993" (UID: "7542d266-7f8c-474a-a8dc-9848376e3993"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:44.775526 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.775509    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7542d266-7f8c-474a-a8dc-9848376e3993-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7542d266-7f8c-474a-a8dc-9848376e3993-istio-podinfo\") pod \"7542d266-7f8c-474a-a8dc-9848376e3993\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") "
Nov 02 02:02:44.775699 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.775558    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7542d266-7f8c-474a-a8dc-9848376e3993-istio-token\" (UniqueName: \"kubernetes.io/projected/7542d266-7f8c-474a-a8dc-9848376e3993-istio-token\") pod \"7542d266-7f8c-474a-a8dc-9848376e3993\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") "
Nov 02 02:02:44.775699 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.775612    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-istio-envoy\") pod \"7542d266-7f8c-474a-a8dc-9848376e3993\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") "
Nov 02 02:02:44.775699 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.775651    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7542d266-7f8c-474a-a8dc-9848376e3993-kube-api-access-2xdcp\" (UniqueName: \"kubernetes.io/projected/7542d266-7f8c-474a-a8dc-9848376e3993-kube-api-access-2xdcp\") pod \"7542d266-7f8c-474a-a8dc-9848376e3993\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") "
Nov 02 02:02:44.775876 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.775671    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-workload-certs" pod "7542d266-7f8c-474a-a8dc-9848376e3993" (UID: "7542d266-7f8c-474a-a8dc-9848376e3993"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:44.775876 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.775707    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7542d266-7f8c-474a-a8dc-9848376e3993-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7542d266-7f8c-474a-a8dc-9848376e3993-istiod-ca-cert\") pod \"7542d266-7f8c-474a-a8dc-9848376e3993\" (UID: \"7542d266-7f8c-474a-a8dc-9848376e3993\") "
Nov 02 02:02:44.775876 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.775747    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-credential-socket" pod "7542d266-7f8c-474a-a8dc-9848376e3993" (UID: "7542d266-7f8c-474a-a8dc-9848376e3993"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:44.775876 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.775766    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-workload-socket" pod "7542d266-7f8c-474a-a8dc-9848376e3993" (UID: "7542d266-7f8c-474a-a8dc-9848376e3993"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:44.776107 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.776079    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7542d266-7f8c-474a-a8dc-9848376e3993-istiod-ca-cert" pod "7542d266-7f8c-474a-a8dc-9848376e3993" (UID: "7542d266-7f8c-474a-a8dc-9848376e3993"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:44.776474 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.776409    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:44.776474 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.776429    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7542d266-7f8c-474a-a8dc-9848376e3993-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:44.776474 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.776438    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:44.776474 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.776446    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:44.776474 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.776455    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:44.777837 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.777812    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7542d266-7f8c-474a-a8dc-9848376e3993-istio-token" pod "7542d266-7f8c-474a-a8dc-9848376e3993" (UID: "7542d266-7f8c-474a-a8dc-9848376e3993"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:44.777987 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.777965    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7542d266-7f8c-474a-a8dc-9848376e3993-kube-api-access-2xdcp" pod "7542d266-7f8c-474a-a8dc-9848376e3993" (UID: "7542d266-7f8c-474a-a8dc-9848376e3993"). InnerVolumeSpecName "kube-api-access-2xdcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:44.778193 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.778177    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-istio-envoy" pod "7542d266-7f8c-474a-a8dc-9848376e3993" (UID: "7542d266-7f8c-474a-a8dc-9848376e3993"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:44.778250 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.778192    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7542d266-7f8c-474a-a8dc-9848376e3993-istio-podinfo" pod "7542d266-7f8c-474a-a8dc-9848376e3993" (UID: "7542d266-7f8c-474a-a8dc-9848376e3993"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:02:44.877798 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.877766    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7542d266-7f8c-474a-a8dc-9848376e3993-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:44.877798 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.877794    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xdcp\" (UniqueName: \"kubernetes.io/projected/7542d266-7f8c-474a-a8dc-9848376e3993-kube-api-access-2xdcp\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:44.877798 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.877806    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7542d266-7f8c-474a-a8dc-9848376e3993-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:44.878432 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:44.877816    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7542d266-7f8c-474a-a8dc-9848376e3993-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:45.638554 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:45.638526    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"
Nov 02 02:02:45.638748 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:45.638564    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs" event={"ID":"7542d266-7f8c-474a-a8dc-9848376e3993","Type":"ContainerDied","Data":"0ce62ce0ffb5367e94cdbc3dd41d5032fd1eb1dbbe68dba4fe3681349b224cf1"}
Nov 02 02:02:45.638748 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:45.638626    5639 scope.go:122] "RemoveContainer" containerID="0601eb1677436e92301b075d8cebd6c527170d03ffaa0895b231b2962dac6305"
Nov 02 02:02:45.855797 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:45.855758    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"]
Nov 02 02:02:45.905539 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:45.905458    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-218802-openshift-default-5d49fbf95-8plrs"]
Nov 02 02:02:46.551823 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:46.551790    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm"]
Nov 02 02:02:46.554649 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:46.554625    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bb0464-674mm"]
Nov 02 02:02:46.709456 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:46.709413    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92" podUID="94012b0a-9356-41bc-8235-9a64f11e8e39" containerName="payload-processing" containerID="cri-o://d01fd69e4214f80db61711a21b8d36e9a60858d27e2840eb777215641a6960d9" gracePeriod=25
Nov 02 02:02:46.774078 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:46.774017    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4" podUID="8ad3b96d-b251-48d6-8b20-5e6f73f48448" containerName="payload-pre-processing" containerID="cri-o://62707b9e9f128e0021f2e2002904b0eea907694eb405e51f3850c383eb8159fb" gracePeriod=25
Nov 02 02:02:47.078824 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.078792    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92"
Nov 02 02:02:47.101795 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.101772    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4"
Nov 02 02:02:47.208367 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.208264    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94012b0a-9356-41bc-8235-9a64f11e8e39-kube-api-access-8spjh\" (UniqueName: \"kubernetes.io/projected/94012b0a-9356-41bc-8235-9a64f11e8e39-kube-api-access-8spjh\") pod \"94012b0a-9356-41bc-8235-9a64f11e8e39\" (UID: \"94012b0a-9356-41bc-8235-9a64f11e8e39\") "
Nov 02 02:02:47.208367 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.208326    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ad3b96d-b251-48d6-8b20-5e6f73f48448-kube-api-access-97rh8\" (UniqueName: \"kubernetes.io/projected/8ad3b96d-b251-48d6-8b20-5e6f73f48448-kube-api-access-97rh8\") pod \"8ad3b96d-b251-48d6-8b20-5e6f73f48448\" (UID: \"8ad3b96d-b251-48d6-8b20-5e6f73f48448\") "
Nov 02 02:02:47.208367 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.208358    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ad3b96d-b251-48d6-8b20-5e6f73f48448-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ad3b96d-b251-48d6-8b20-5e6f73f48448-plugins-config-volume\") pod \"8ad3b96d-b251-48d6-8b20-5e6f73f48448\" (UID: \"8ad3b96d-b251-48d6-8b20-5e6f73f48448\") "
Nov 02 02:02:47.208653 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.208592    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94012b0a-9356-41bc-8235-9a64f11e8e39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94012b0a-9356-41bc-8235-9a64f11e8e39-plugins-config-volume\") pod \"94012b0a-9356-41bc-8235-9a64f11e8e39\" (UID: \"94012b0a-9356-41bc-8235-9a64f11e8e39\") "
Nov 02 02:02:47.208818 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.208785    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ad3b96d-b251-48d6-8b20-5e6f73f48448-plugins-config-volume" pod "8ad3b96d-b251-48d6-8b20-5e6f73f48448" (UID: "8ad3b96d-b251-48d6-8b20-5e6f73f48448"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:47.208952 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.208865    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94012b0a-9356-41bc-8235-9a64f11e8e39-plugins-config-volume" pod "94012b0a-9356-41bc-8235-9a64f11e8e39" (UID: "94012b0a-9356-41bc-8235-9a64f11e8e39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:47.209441 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.209417    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ad3b96d-b251-48d6-8b20-5e6f73f48448-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:47.209578 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.209446    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94012b0a-9356-41bc-8235-9a64f11e8e39-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:47.210899 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.210878    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad3b96d-b251-48d6-8b20-5e6f73f48448-kube-api-access-97rh8" pod "8ad3b96d-b251-48d6-8b20-5e6f73f48448" (UID: "8ad3b96d-b251-48d6-8b20-5e6f73f48448"). InnerVolumeSpecName "kube-api-access-97rh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:47.211037 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.211018    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94012b0a-9356-41bc-8235-9a64f11e8e39-kube-api-access-8spjh" pod "94012b0a-9356-41bc-8235-9a64f11e8e39" (UID: "94012b0a-9356-41bc-8235-9a64f11e8e39"). InnerVolumeSpecName "kube-api-access-8spjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:47.220221 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.220195    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a51de5a-8641-47b0-be51-21d1be052318" path="/var/lib/kubelet/pods/4a51de5a-8641-47b0-be51-21d1be052318/volumes"
Nov 02 02:02:47.220786 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.220766    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7542d266-7f8c-474a-a8dc-9848376e3993" path="/var/lib/kubelet/pods/7542d266-7f8c-474a-a8dc-9848376e3993/volumes"
Nov 02 02:02:47.310870 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.310834    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8spjh\" (UniqueName: \"kubernetes.io/projected/94012b0a-9356-41bc-8235-9a64f11e8e39-kube-api-access-8spjh\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:47.310870 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.310860    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97rh8\" (UniqueName: \"kubernetes.io/projected/8ad3b96d-b251-48d6-8b20-5e6f73f48448-kube-api-access-97rh8\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:47.654203 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.654172    5639 generic.go:356] "Generic (PLEG): container finished" podID="8ad3b96d-b251-48d6-8b20-5e6f73f48448" containerID="62707b9e9f128e0021f2e2002904b0eea907694eb405e51f3850c383eb8159fb" exitCode=0
Nov 02 02:02:47.654413 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.654216    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4"
Nov 02 02:02:47.654413 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.654260    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4" event={"ID":"8ad3b96d-b251-48d6-8b20-5e6f73f48448","Type":"ContainerDied","Data":"62707b9e9f128e0021f2e2002904b0eea907694eb405e51f3850c383eb8159fb"}
Nov 02 02:02:47.654413 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.654336    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4" event={"ID":"8ad3b96d-b251-48d6-8b20-5e6f73f48448","Type":"ContainerDied","Data":"1a2015c57b80263860016a025b1e7d26627a82db6def9448ac805bc506d1bec8"}
Nov 02 02:02:47.654413 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.654363    5639 scope.go:122] "RemoveContainer" containerID="62707b9e9f128e0021f2e2002904b0eea907694eb405e51f3850c383eb8159fb"
Nov 02 02:02:47.655991 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.655968    5639 generic.go:356] "Generic (PLEG): container finished" podID="94012b0a-9356-41bc-8235-9a64f11e8e39" containerID="d01fd69e4214f80db61711a21b8d36e9a60858d27e2840eb777215641a6960d9" exitCode=0
Nov 02 02:02:47.656120 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.656007    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92"
Nov 02 02:02:47.656120 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.656024    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92" event={"ID":"94012b0a-9356-41bc-8235-9a64f11e8e39","Type":"ContainerDied","Data":"d01fd69e4214f80db61711a21b8d36e9a60858d27e2840eb777215641a6960d9"}
Nov 02 02:02:47.656120 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.656065    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92" event={"ID":"94012b0a-9356-41bc-8235-9a64f11e8e39","Type":"ContainerDied","Data":"ffe13f28eb6d481a53fd6bf3903313a004bf4dcc539c322a57bfd8482b814dfb"}
Nov 02 02:02:47.665898 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.665875    5639 scope.go:122] "RemoveContainer" containerID="62707b9e9f128e0021f2e2002904b0eea907694eb405e51f3850c383eb8159fb"
Nov 02 02:02:47.666170 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:02:47.666147    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62707b9e9f128e0021f2e2002904b0eea907694eb405e51f3850c383eb8159fb\": container with ID starting with 62707b9e9f128e0021f2e2002904b0eea907694eb405e51f3850c383eb8159fb not found: ID does not exist" containerID="62707b9e9f128e0021f2e2002904b0eea907694eb405e51f3850c383eb8159fb"
Nov 02 02:02:47.666259 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.666182    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62707b9e9f128e0021f2e2002904b0eea907694eb405e51f3850c383eb8159fb"} err="failed to get container status \"62707b9e9f128e0021f2e2002904b0eea907694eb405e51f3850c383eb8159fb\": rpc error: code = NotFound desc = could not find container \"62707b9e9f128e0021f2e2002904b0eea907694eb405e51f3850c383eb8159fb\": container with ID starting with 62707b9e9f128e0021f2e2002904b0eea907694eb405e51f3850c383eb8159fb not found: ID does not exist"
Nov 02 02:02:47.666259 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.666207    5639 scope.go:122] "RemoveContainer" containerID="d01fd69e4214f80db61711a21b8d36e9a60858d27e2840eb777215641a6960d9"
Nov 02 02:02:47.675939 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.675914    5639 scope.go:122] "RemoveContainer" containerID="d01fd69e4214f80db61711a21b8d36e9a60858d27e2840eb777215641a6960d9"
Nov 02 02:02:47.676251 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:02:47.676230    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d01fd69e4214f80db61711a21b8d36e9a60858d27e2840eb777215641a6960d9\": container with ID starting with d01fd69e4214f80db61711a21b8d36e9a60858d27e2840eb777215641a6960d9 not found: ID does not exist" containerID="d01fd69e4214f80db61711a21b8d36e9a60858d27e2840eb777215641a6960d9"
Nov 02 02:02:47.676321 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.676260    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d01fd69e4214f80db61711a21b8d36e9a60858d27e2840eb777215641a6960d9"} err="failed to get container status \"d01fd69e4214f80db61711a21b8d36e9a60858d27e2840eb777215641a6960d9\": rpc error: code = NotFound desc = could not find container \"d01fd69e4214f80db61711a21b8d36e9a60858d27e2840eb777215641a6960d9\": container with ID starting with d01fd69e4214f80db61711a21b8d36e9a60858d27e2840eb777215641a6960d9 not found: ID does not exist"
Nov 02 02:02:47.819887 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.819853    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4"]
Nov 02 02:02:47.824765 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:47.824735    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bb0464-684b8c84c9-xlcp4"]
Nov 02 02:02:48.018648 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:48.018569    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92"]
Nov 02 02:02:48.046008 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:48.045975    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-bb0464-87df6bdc9-q7p92"]
Nov 02 02:02:49.220475 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:49.220442    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ad3b96d-b251-48d6-8b20-5e6f73f48448" path="/var/lib/kubelet/pods/8ad3b96d-b251-48d6-8b20-5e6f73f48448/volumes"
Nov 02 02:02:49.220918 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:49.220904    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94012b0a-9356-41bc-8235-9a64f11e8e39" path="/var/lib/kubelet/pods/94012b0a-9356-41bc-8235-9a64f11e8e39/volumes"
Nov 02 02:02:50.044135 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:50.044097    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg"]
Nov 02 02:02:50.093359 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:50.093320    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-19436b-kk2fg"]
Nov 02 02:02:51.219073 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:51.219040    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40d80410-2d8b-4eb4-b3c3-98b90b3b60b6" path="/var/lib/kubelet/pods/40d80410-2d8b-4eb4-b3c3-98b90b3b60b6/volumes"
Nov 02 02:02:51.746906 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:51.746865    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw"]
Nov 02 02:02:51.858833 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:51.858796    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb"]
Nov 02 02:02:53.936038 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:53.936005    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"]
Nov 02 02:02:53.936505 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:53.936228    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd" podUID="124b9d97-7aec-44dd-a24b-06be952004fc" containerName="istio-proxy" containerID="cri-o://f35931641094e1695c3108bb58b7c40445d8c2ffafa56c4180fe6d6280b14bd7" gracePeriod=30
Nov 02 02:02:54.935755 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:54.935720    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9"]
Nov 02 02:02:54.936468 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:54.936446    5639 cpu_manager.go:414] "containerMap: removing container" podUID="8ad3b96d-b251-48d6-8b20-5e6f73f48448" containerName="payload-pre-processing"
Nov 02 02:02:54.936468 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:54.936467    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ad3b96d-b251-48d6-8b20-5e6f73f48448" containerName="payload-pre-processing"
Nov 02 02:02:54.936992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:54.936489    5639 cpu_manager.go:414] "containerMap: removing container" podUID="94012b0a-9356-41bc-8235-9a64f11e8e39" containerName="payload-processing"
Nov 02 02:02:54.936992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:54.936494    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94012b0a-9356-41bc-8235-9a64f11e8e39" containerName="payload-processing"
Nov 02 02:02:54.936992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:54.936517    5639 cpu_manager.go:414] "containerMap: removing container" podUID="7542d266-7f8c-474a-a8dc-9848376e3993" containerName="istio-proxy"
Nov 02 02:02:54.936992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:54.936525    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7542d266-7f8c-474a-a8dc-9848376e3993" containerName="istio-proxy"
Nov 02 02:02:54.936992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:54.936546    5639 cpu_manager.go:414] "containerMap: removing container" podUID="62180ece-d5f2-48a4-8b9f-7cfe4e814121" containerName="revoke-keys"
Nov 02 02:02:54.936992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:54.936551    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62180ece-d5f2-48a4-8b9f-7cfe4e814121" containerName="revoke-keys"
Nov 02 02:02:54.936992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:54.936663    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="7542d266-7f8c-474a-a8dc-9848376e3993" containerName="istio-proxy"
Nov 02 02:02:54.936992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:54.936675    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ad3b96d-b251-48d6-8b20-5e6f73f48448" containerName="payload-pre-processing"
Nov 02 02:02:54.936992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:54.936686    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="94012b0a-9356-41bc-8235-9a64f11e8e39" containerName="payload-processing"
Nov 02 02:02:54.936992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:54.936699    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="62180ece-d5f2-48a4-8b9f-7cfe4e814121" containerName="revoke-keys"
Nov 02 02:02:54.942259 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:54.942239    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9"
Nov 02 02:02:54.992924 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:54.992897    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9"]
Nov 02 02:02:55.038351 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:55.038316    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:02:55.038351 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:55.038344    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:02:55.038590 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:55.038391    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:02:55.038590 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:55.038444    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:02:55.056689 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:55.056655    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad1ef4a4-1194-4dc5-bd7c-836717fda66e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9\" (UID: \"ad1ef4a4-1194-4dc5-bd7c-836717fda66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9"
Nov 02 02:02:55.056850 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:55.056815    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad1ef4a4-1194-4dc5-bd7c-836717fda66e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9\" (UID: \"ad1ef4a4-1194-4dc5-bd7c-836717fda66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9"
Nov 02 02:02:55.056972 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:55.056950    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqbds\" (UniqueName: \"kubernetes.io/projected/ad1ef4a4-1194-4dc5-bd7c-836717fda66e-kube-api-access-wqbds\") pod \"maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9\" (UID: \"ad1ef4a4-1194-4dc5-bd7c-836717fda66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9"
Nov 02 02:02:55.157922 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:55.157886    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad1ef4a4-1194-4dc5-bd7c-836717fda66e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9\" (UID: \"ad1ef4a4-1194-4dc5-bd7c-836717fda66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9"
Nov 02 02:02:55.158120 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:55.157997    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad1ef4a4-1194-4dc5-bd7c-836717fda66e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9\" (UID: \"ad1ef4a4-1194-4dc5-bd7c-836717fda66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9"
Nov 02 02:02:55.158120 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:55.158077    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqbds\" (UniqueName: \"kubernetes.io/projected/ad1ef4a4-1194-4dc5-bd7c-836717fda66e-kube-api-access-wqbds\") pod \"maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9\" (UID: \"ad1ef4a4-1194-4dc5-bd7c-836717fda66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9"
Nov 02 02:02:55.158626 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:55.158601    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad1ef4a4-1194-4dc5-bd7c-836717fda66e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9\" (UID: \"ad1ef4a4-1194-4dc5-bd7c-836717fda66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9"
Nov 02 02:02:55.160704 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:55.160678    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad1ef4a4-1194-4dc5-bd7c-836717fda66e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9\" (UID: \"ad1ef4a4-1194-4dc5-bd7c-836717fda66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9"
Nov 02 02:02:55.219192 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:55.219126    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqbds\" (UniqueName: \"kubernetes.io/projected/ad1ef4a4-1194-4dc5-bd7c-836717fda66e-kube-api-access-wqbds\") pod \"maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9\" (UID: \"ad1ef4a4-1194-4dc5-bd7c-836717fda66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9"
Nov 02 02:02:55.256229 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:55.256202    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9"
Nov 02 02:02:55.633866 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:55.633836    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9"]
Nov 02 02:02:55.635389 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:02:55.635356    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad1ef4a4_1194_4dc5_bd7c_836717fda66e.slice/crio-d64a5353094918e924d7c32fa3e06427ee9eae42053d1eb0a07d25af4b976a76 WatchSource:0}: Error finding container d64a5353094918e924d7c32fa3e06427ee9eae42053d1eb0a07d25af4b976a76: Status 404 returned error can't find the container with id d64a5353094918e924d7c32fa3e06427ee9eae42053d1eb0a07d25af4b976a76
Nov 02 02:02:55.728036 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:55.728001    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9" event={"ID":"ad1ef4a4-1194-4dc5-bd7c-836717fda66e","Type":"ContainerStarted","Data":"d64a5353094918e924d7c32fa3e06427ee9eae42053d1eb0a07d25af4b976a76"}
Nov 02 02:02:56.739667 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:56.739626    5639 generic.go:356] "Generic (PLEG): container finished" podID="ad1ef4a4-1194-4dc5-bd7c-836717fda66e" containerID="9e3b83f00646fa88eb93d86ecc55f1a7ba104129393bd0eb903d945c4398b24d" exitCode=0
Nov 02 02:02:56.740132 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:56.739750    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9" event={"ID":"ad1ef4a4-1194-4dc5-bd7c-836717fda66e","Type":"ContainerDied","Data":"9e3b83f00646fa88eb93d86ecc55f1a7ba104129393bd0eb903d945c4398b24d"}
Nov 02 02:02:56.748077 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:56.748042    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw" podUID="4b1f54d2-0cbb-42a0-96fd-1fea08daae41" containerName="payload-processing" containerID="cri-o://d95d5a50ddc20b0864b248f84bbbf98865c067b18539f05818471e8487097117" gracePeriod=25
Nov 02 02:02:56.777358 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:56.777253    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"]
Nov 02 02:02:56.777524 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:56.777501    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx" podUID="5d8da016-d523-4efd-91de-aae563f7f1a6" containerName="istio-proxy" containerID="cri-o://4bbc475bcfb487be3b93e77093d30391682727e237959b009b40910ad2bd6f4a" gracePeriod=30
Nov 02 02:02:56.876306 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:56.871083    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb" podUID="1b20cafe-5759-403c-b4f8-ed895c71de18" containerName="payload-pre-processing" containerID="cri-o://71b2111c5fa7d05e012028a59bf0db0d080604cfef9eefd3cc784cf37f8f17d9" gracePeriod=25
Nov 02 02:02:57.093699 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.093669    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw"
Nov 02 02:02:57.185143 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.185113    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b1f54d2-0cbb-42a0-96fd-1fea08daae41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1f54d2-0cbb-42a0-96fd-1fea08daae41-plugins-config-volume\") pod \"4b1f54d2-0cbb-42a0-96fd-1fea08daae41\" (UID: \"4b1f54d2-0cbb-42a0-96fd-1fea08daae41\") "
Nov 02 02:02:57.185328 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.185159    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b1f54d2-0cbb-42a0-96fd-1fea08daae41-kube-api-access-xstrm\" (UniqueName: \"kubernetes.io/projected/4b1f54d2-0cbb-42a0-96fd-1fea08daae41-kube-api-access-xstrm\") pod \"4b1f54d2-0cbb-42a0-96fd-1fea08daae41\" (UID: \"4b1f54d2-0cbb-42a0-96fd-1fea08daae41\") "
Nov 02 02:02:57.185509 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.185486    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b1f54d2-0cbb-42a0-96fd-1fea08daae41-plugins-config-volume" pod "4b1f54d2-0cbb-42a0-96fd-1fea08daae41" (UID: "4b1f54d2-0cbb-42a0-96fd-1fea08daae41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:57.185640 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.185622    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1f54d2-0cbb-42a0-96fd-1fea08daae41-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:57.187470 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.187446    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1f54d2-0cbb-42a0-96fd-1fea08daae41-kube-api-access-xstrm" pod "4b1f54d2-0cbb-42a0-96fd-1fea08daae41" (UID: "4b1f54d2-0cbb-42a0-96fd-1fea08daae41"). InnerVolumeSpecName "kube-api-access-xstrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:57.209880 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.209852    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb"
Nov 02 02:02:57.286889 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.286853    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b20cafe-5759-403c-b4f8-ed895c71de18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b20cafe-5759-403c-b4f8-ed895c71de18-plugins-config-volume\") pod \"1b20cafe-5759-403c-b4f8-ed895c71de18\" (UID: \"1b20cafe-5759-403c-b4f8-ed895c71de18\") "
Nov 02 02:02:57.287092 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.286915    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b20cafe-5759-403c-b4f8-ed895c71de18-kube-api-access-qjgf6\" (UniqueName: \"kubernetes.io/projected/1b20cafe-5759-403c-b4f8-ed895c71de18-kube-api-access-qjgf6\") pod \"1b20cafe-5759-403c-b4f8-ed895c71de18\" (UID: \"1b20cafe-5759-403c-b4f8-ed895c71de18\") "
Nov 02 02:02:57.287241 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.287213    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b20cafe-5759-403c-b4f8-ed895c71de18-plugins-config-volume" pod "1b20cafe-5759-403c-b4f8-ed895c71de18" (UID: "1b20cafe-5759-403c-b4f8-ed895c71de18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:57.287368 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.287355    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b20cafe-5759-403c-b4f8-ed895c71de18-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:57.287410 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.287371    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xstrm\" (UniqueName: \"kubernetes.io/projected/4b1f54d2-0cbb-42a0-96fd-1fea08daae41-kube-api-access-xstrm\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:57.289200 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.289176    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b20cafe-5759-403c-b4f8-ed895c71de18-kube-api-access-qjgf6" pod "1b20cafe-5759-403c-b4f8-ed895c71de18" (UID: "1b20cafe-5759-403c-b4f8-ed895c71de18"). InnerVolumeSpecName "kube-api-access-qjgf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:57.388868 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.388833    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjgf6\" (UniqueName: \"kubernetes.io/projected/1b20cafe-5759-403c-b4f8-ed895c71de18-kube-api-access-qjgf6\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:57.421738 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:02:57.421706    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:02:57.649891 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.649854    5639 patch_prober.go:29] interesting pod/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:02:57.650094 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.649929    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx" podUID="5d8da016-d523-4efd-91de-aae563f7f1a6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:02:57.749581 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.749545    5639 generic.go:356] "Generic (PLEG): container finished" podID="1b20cafe-5759-403c-b4f8-ed895c71de18" containerID="71b2111c5fa7d05e012028a59bf0db0d080604cfef9eefd3cc784cf37f8f17d9" exitCode=0
Nov 02 02:02:57.750019 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.749592    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb"
Nov 02 02:02:57.750019 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.749625    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb" event={"ID":"1b20cafe-5759-403c-b4f8-ed895c71de18","Type":"ContainerDied","Data":"71b2111c5fa7d05e012028a59bf0db0d080604cfef9eefd3cc784cf37f8f17d9"}
Nov 02 02:02:57.750019 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.749672    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb" event={"ID":"1b20cafe-5759-403c-b4f8-ed895c71de18","Type":"ContainerDied","Data":"8812b6c1186a6da16fe57aed803272b920d6cf06442c9e0bf8bae690cb96f9cb"}
Nov 02 02:02:57.750019 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.749702    5639 scope.go:122] "RemoveContainer" containerID="71b2111c5fa7d05e012028a59bf0db0d080604cfef9eefd3cc784cf37f8f17d9"
Nov 02 02:02:57.751383 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.751360    5639 generic.go:356] "Generic (PLEG): container finished" podID="4b1f54d2-0cbb-42a0-96fd-1fea08daae41" containerID="d95d5a50ddc20b0864b248f84bbbf98865c067b18539f05818471e8487097117" exitCode=0
Nov 02 02:02:57.751534 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.751399    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw"
Nov 02 02:02:57.751534 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.751512    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw" event={"ID":"4b1f54d2-0cbb-42a0-96fd-1fea08daae41","Type":"ContainerDied","Data":"d95d5a50ddc20b0864b248f84bbbf98865c067b18539f05818471e8487097117"}
Nov 02 02:02:57.751652 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.751566    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw" event={"ID":"4b1f54d2-0cbb-42a0-96fd-1fea08daae41","Type":"ContainerDied","Data":"4623f12c4cdfe42cb06d0262f3730e465aecc2ddfcd708793dc481751fd7afb0"}
Nov 02 02:02:57.760801 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.760782    5639 scope.go:122] "RemoveContainer" containerID="71b2111c5fa7d05e012028a59bf0db0d080604cfef9eefd3cc784cf37f8f17d9"
Nov 02 02:02:57.761099 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:02:57.761073    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71b2111c5fa7d05e012028a59bf0db0d080604cfef9eefd3cc784cf37f8f17d9\": container with ID starting with 71b2111c5fa7d05e012028a59bf0db0d080604cfef9eefd3cc784cf37f8f17d9 not found: ID does not exist" containerID="71b2111c5fa7d05e012028a59bf0db0d080604cfef9eefd3cc784cf37f8f17d9"
Nov 02 02:02:57.761196 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.761108    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b2111c5fa7d05e012028a59bf0db0d080604cfef9eefd3cc784cf37f8f17d9"} err="failed to get container status \"71b2111c5fa7d05e012028a59bf0db0d080604cfef9eefd3cc784cf37f8f17d9\": rpc error: code = NotFound desc = could not find container \"71b2111c5fa7d05e012028a59bf0db0d080604cfef9eefd3cc784cf37f8f17d9\": container with ID starting with 71b2111c5fa7d05e012028a59bf0db0d080604cfef9eefd3cc784cf37f8f17d9 not found: ID does not exist"
Nov 02 02:02:57.761196 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.761128    5639 scope.go:122] "RemoveContainer" containerID="d95d5a50ddc20b0864b248f84bbbf98865c067b18539f05818471e8487097117"
Nov 02 02:02:57.772386 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.772366    5639 scope.go:122] "RemoveContainer" containerID="d95d5a50ddc20b0864b248f84bbbf98865c067b18539f05818471e8487097117"
Nov 02 02:02:57.772744 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:02:57.772720    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d95d5a50ddc20b0864b248f84bbbf98865c067b18539f05818471e8487097117\": container with ID starting with d95d5a50ddc20b0864b248f84bbbf98865c067b18539f05818471e8487097117 not found: ID does not exist" containerID="d95d5a50ddc20b0864b248f84bbbf98865c067b18539f05818471e8487097117"
Nov 02 02:02:57.772815 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.772753    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d95d5a50ddc20b0864b248f84bbbf98865c067b18539f05818471e8487097117"} err="failed to get container status \"d95d5a50ddc20b0864b248f84bbbf98865c067b18539f05818471e8487097117\": rpc error: code = NotFound desc = could not find container \"d95d5a50ddc20b0864b248f84bbbf98865c067b18539f05818471e8487097117\": container with ID starting with d95d5a50ddc20b0864b248f84bbbf98865c067b18539f05818471e8487097117 not found: ID does not exist"
Nov 02 02:02:57.893716 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.893692    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9"
Nov 02 02:02:57.914876 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.914797    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb"]
Nov 02 02:02:57.917839 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.917818    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5a72-566b564f6c-58nsb"]
Nov 02 02:02:57.995765 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.995737    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad1ef4a4-1194-4dc5-bd7c-836717fda66e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad1ef4a4-1194-4dc5-bd7c-836717fda66e-maas-api-openshift-ca\") pod \"ad1ef4a4-1194-4dc5-bd7c-836717fda66e\" (UID: \"ad1ef4a4-1194-4dc5-bd7c-836717fda66e\") "
Nov 02 02:02:57.995938 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.995833    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad1ef4a4-1194-4dc5-bd7c-836717fda66e-kube-api-access-wqbds\" (UniqueName: \"kubernetes.io/projected/ad1ef4a4-1194-4dc5-bd7c-836717fda66e-kube-api-access-wqbds\") pod \"ad1ef4a4-1194-4dc5-bd7c-836717fda66e\" (UID: \"ad1ef4a4-1194-4dc5-bd7c-836717fda66e\") "
Nov 02 02:02:57.995938 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.995879    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad1ef4a4-1194-4dc5-bd7c-836717fda66e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad1ef4a4-1194-4dc5-bd7c-836717fda66e-maas-api-xks-ca\") pod \"ad1ef4a4-1194-4dc5-bd7c-836717fda66e\" (UID: \"ad1ef4a4-1194-4dc5-bd7c-836717fda66e\") "
Nov 02 02:02:57.996167 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.996135    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad1ef4a4-1194-4dc5-bd7c-836717fda66e-maas-api-openshift-ca" pod "ad1ef4a4-1194-4dc5-bd7c-836717fda66e" (UID: "ad1ef4a4-1194-4dc5-bd7c-836717fda66e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:57.996440 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.996423    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad1ef4a4-1194-4dc5-bd7c-836717fda66e-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:57.998104 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.998058    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad1ef4a4-1194-4dc5-bd7c-836717fda66e-maas-api-xks-ca" pod "ad1ef4a4-1194-4dc5-bd7c-836717fda66e" (UID: "ad1ef4a4-1194-4dc5-bd7c-836717fda66e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:02:57.998104 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:57.998090    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad1ef4a4-1194-4dc5-bd7c-836717fda66e-kube-api-access-wqbds" pod "ad1ef4a4-1194-4dc5-bd7c-836717fda66e" (UID: "ad1ef4a4-1194-4dc5-bd7c-836717fda66e"). InnerVolumeSpecName "kube-api-access-wqbds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:58.025617 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.025588    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw"]
Nov 02 02:02:58.030379 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.030349    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ef5a72-68cf8984bf-9sqqw"]
Nov 02 02:02:58.097434 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.097398    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqbds\" (UniqueName: \"kubernetes.io/projected/ad1ef4a4-1194-4dc5-bd7c-836717fda66e-kube-api-access-wqbds\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:58.097434 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.097428    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad1ef4a4-1194-4dc5-bd7c-836717fda66e-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:58.174012 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.173928    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"]
Nov 02 02:02:58.174922 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.174895    5639 cpu_manager.go:414] "containerMap: removing container" podUID="ad1ef4a4-1194-4dc5-bd7c-836717fda66e" containerName="revoke-keys"
Nov 02 02:02:58.174922 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.174920    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad1ef4a4-1194-4dc5-bd7c-836717fda66e" containerName="revoke-keys"
Nov 02 02:02:58.175135 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.174939    5639 cpu_manager.go:414] "containerMap: removing container" podUID="4b1f54d2-0cbb-42a0-96fd-1fea08daae41" containerName="payload-processing"
Nov 02 02:02:58.175135 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.174948    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1f54d2-0cbb-42a0-96fd-1fea08daae41" containerName="payload-processing"
Nov 02 02:02:58.175135 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.174967    5639 cpu_manager.go:414] "containerMap: removing container" podUID="1b20cafe-5759-403c-b4f8-ed895c71de18" containerName="payload-pre-processing"
Nov 02 02:02:58.175135 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.174976    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b20cafe-5759-403c-b4f8-ed895c71de18" containerName="payload-pre-processing"
Nov 02 02:02:58.175135 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.174995    5639 cpu_manager.go:414] "containerMap: removing container" podUID="7542d266-7f8c-474a-a8dc-9848376e3993" containerName="istio-proxy"
Nov 02 02:02:58.175135 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.175002    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7542d266-7f8c-474a-a8dc-9848376e3993" containerName="istio-proxy"
Nov 02 02:02:58.175556 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.175153    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad1ef4a4-1194-4dc5-bd7c-836717fda66e" containerName="revoke-keys"
Nov 02 02:02:58.175556 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.175170    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b1f54d2-0cbb-42a0-96fd-1fea08daae41" containerName="payload-processing"
Nov 02 02:02:58.175556 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.175183    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b20cafe-5759-403c-b4f8-ed895c71de18" containerName="payload-pre-processing"
Nov 02 02:02:58.175556 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.175196    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="7542d266-7f8c-474a-a8dc-9848376e3993" containerName="istio-proxy"
Nov 02 02:02:58.179020 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.178997    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.289359 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.260460    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0c954d-openshift-default-dockercfg-r2v8n\""
Nov 02 02:02:58.289359 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.264317    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"]
Nov 02 02:02:58.310128 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.310084    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-istio-data\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.310356 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.310211    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-workload-socket\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.310476 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.310384    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-istio-envoy\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.310590 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.310569    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-credential-socket\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.310687 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.310611    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-workload-certs\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.310752 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.310713    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qcpn\" (UniqueName: \"kubernetes.io/projected/dae611da-8694-4348-9dff-90bd553e95ec-kube-api-access-4qcpn\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.310816 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.310777    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae611da-8694-4348-9dff-90bd553e95ec-istiod-ca-cert\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.310874 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.310858    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae611da-8694-4348-9dff-90bd553e95ec-istio-podinfo\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.310937 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.310921    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae611da-8694-4348-9dff-90bd553e95ec-istio-token\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.412558 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.412528    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-credential-socket\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.412558 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.412564    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-workload-certs\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.412808 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.412610    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qcpn\" (UniqueName: \"kubernetes.io/projected/dae611da-8694-4348-9dff-90bd553e95ec-kube-api-access-4qcpn\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.412808 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.412644    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae611da-8694-4348-9dff-90bd553e95ec-istiod-ca-cert\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.412808 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.412694    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae611da-8694-4348-9dff-90bd553e95ec-istio-podinfo\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.412808 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.412735    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae611da-8694-4348-9dff-90bd553e95ec-istio-token\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.412808 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.412771    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-istio-data\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.413086 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.412867    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-workload-socket\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.413086 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.412868    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-credential-socket\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.413086 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.412931    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-istio-envoy\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.413086 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.412956    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-workload-certs\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.413345 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.413157    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-workload-socket\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.413345 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.413271    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-istio-data\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.413677 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.413654    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae611da-8694-4348-9dff-90bd553e95ec-istiod-ca-cert\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.415316 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.415271    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-istio-envoy\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.415414 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.415380    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae611da-8694-4348-9dff-90bd553e95ec-istio-podinfo\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.473472 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.473407    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae611da-8694-4348-9dff-90bd553e95ec-istio-token\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.473584 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.473529    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qcpn\" (UniqueName: \"kubernetes.io/projected/dae611da-8694-4348-9dff-90bd553e95ec-kube-api-access-4qcpn\") pod \"e2e-api-a-0c954d-openshift-default-5954667d99-66wvd\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") " pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.497233 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.497204    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:02:58.762536 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.762439    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9" event={"ID":"ad1ef4a4-1194-4dc5-bd7c-836717fda66e","Type":"ContainerDied","Data":"d64a5353094918e924d7c32fa3e06427ee9eae42053d1eb0a07d25af4b976a76"}
Nov 02 02:02:58.762536 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.762468    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9"
Nov 02 02:02:58.762536 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.762501    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d64a5353094918e924d7c32fa3e06427ee9eae42053d1eb0a07d25af4b976a76"
Nov 02 02:02:58.936032 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.936005    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"]
Nov 02 02:02:58.937433 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:02:58.937349    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae611da_8694_4348_9dff_90bd553e95ec.slice/crio-1d5fe982ca290531d79bb367087d96c8dbddbab48b2dafd4e6720f55f83ff38b WatchSource:0}: Error finding container 1d5fe982ca290531d79bb367087d96c8dbddbab48b2dafd4e6720f55f83ff38b: Status 404 returned error can't find the container with id 1d5fe982ca290531d79bb367087d96c8dbddbab48b2dafd4e6720f55f83ff38b
Nov 02 02:02:58.940732 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.940692    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:02:58.940830 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.940783    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:02:58.940875 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:58.940830    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:02:59.222109 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.222081    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b20cafe-5759-403c-b4f8-ed895c71de18" path="/var/lib/kubelet/pods/1b20cafe-5759-403c-b4f8-ed895c71de18/volumes"
Nov 02 02:02:59.222635 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.222615    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b1f54d2-0cbb-42a0-96fd-1fea08daae41" path="/var/lib/kubelet/pods/4b1f54d2-0cbb-42a0-96fd-1fea08daae41/volumes"
Nov 02 02:02:59.248170 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.248145    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 02:02:59.326842 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.326747    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/124b9d97-7aec-44dd-a24b-06be952004fc-istio-token\" (UniqueName: \"kubernetes.io/projected/124b9d97-7aec-44dd-a24b-06be952004fc-istio-token\") pod \"124b9d97-7aec-44dd-a24b-06be952004fc\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") "
Nov 02 02:02:59.326842 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.326798    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/124b9d97-7aec-44dd-a24b-06be952004fc-kube-api-access-jsz48\" (UniqueName: \"kubernetes.io/projected/124b9d97-7aec-44dd-a24b-06be952004fc-kube-api-access-jsz48\") pod \"124b9d97-7aec-44dd-a24b-06be952004fc\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") "
Nov 02 02:02:59.326842 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.326828    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-workload-certs\") pod \"124b9d97-7aec-44dd-a24b-06be952004fc\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") "
Nov 02 02:02:59.327122 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.326908    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/124b9d97-7aec-44dd-a24b-06be952004fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/124b9d97-7aec-44dd-a24b-06be952004fc-istiod-ca-cert\") pod \"124b9d97-7aec-44dd-a24b-06be952004fc\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") "
Nov 02 02:02:59.327122 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.326934    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-istio-envoy\") pod \"124b9d97-7aec-44dd-a24b-06be952004fc\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") "
Nov 02 02:02:59.327122 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.326996    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-workload-socket\") pod \"124b9d97-7aec-44dd-a24b-06be952004fc\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") "
Nov 02 02:02:59.327122 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.327102    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-istio-data\") pod \"124b9d97-7aec-44dd-a24b-06be952004fc\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") "
Nov 02 02:02:59.327393 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.327246    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-credential-socket\") pod \"124b9d97-7aec-44dd-a24b-06be952004fc\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") "
Nov 02 02:02:59.327393 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.327305    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/124b9d97-7aec-44dd-a24b-06be952004fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/124b9d97-7aec-44dd-a24b-06be952004fc-istio-podinfo\") pod \"124b9d97-7aec-44dd-a24b-06be952004fc\" (UID: \"124b9d97-7aec-44dd-a24b-06be952004fc\") "
Nov 02 02:02:59.328391 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.328057    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-workload-certs" pod "124b9d97-7aec-44dd-a24b-06be952004fc" (UID: "124b9d97-7aec-44dd-a24b-06be952004fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:59.328672 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.328109    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/124b9d97-7aec-44dd-a24b-06be952004fc-istiod-ca-cert" pod "124b9d97-7aec-44dd-a24b-06be952004fc" (UID: "124b9d97-7aec-44dd-a24b-06be952004fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:59.328672 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.328590    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-credential-socket" pod "124b9d97-7aec-44dd-a24b-06be952004fc" (UID: "124b9d97-7aec-44dd-a24b-06be952004fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:59.328814 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.328772    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-workload-socket" pod "124b9d97-7aec-44dd-a24b-06be952004fc" (UID: "124b9d97-7aec-44dd-a24b-06be952004fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:59.328946 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.328921    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-istio-data" pod "124b9d97-7aec-44dd-a24b-06be952004fc" (UID: "124b9d97-7aec-44dd-a24b-06be952004fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:59.329653 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.329628    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124b9d97-7aec-44dd-a24b-06be952004fc-kube-api-access-jsz48" pod "124b9d97-7aec-44dd-a24b-06be952004fc" (UID: "124b9d97-7aec-44dd-a24b-06be952004fc"). InnerVolumeSpecName "kube-api-access-jsz48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:59.329754 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.329675    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124b9d97-7aec-44dd-a24b-06be952004fc-istio-token" pod "124b9d97-7aec-44dd-a24b-06be952004fc" (UID: "124b9d97-7aec-44dd-a24b-06be952004fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:59.330099 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.330072    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/124b9d97-7aec-44dd-a24b-06be952004fc-istio-podinfo" pod "124b9d97-7aec-44dd-a24b-06be952004fc" (UID: "124b9d97-7aec-44dd-a24b-06be952004fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:02:59.330743 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.330721    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-istio-envoy" pod "124b9d97-7aec-44dd-a24b-06be952004fc" (UID: "124b9d97-7aec-44dd-a24b-06be952004fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:59.415729 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.415688    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"]
Nov 02 02:02:59.429918 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.429880    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:59.429918 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.429915    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/124b9d97-7aec-44dd-a24b-06be952004fc-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:59.430108 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.429929    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/124b9d97-7aec-44dd-a24b-06be952004fc-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:59.430108 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.429943    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsz48\" (UniqueName: \"kubernetes.io/projected/124b9d97-7aec-44dd-a24b-06be952004fc-kube-api-access-jsz48\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:59.430108 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.429958    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:59.430108 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.429971    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/124b9d97-7aec-44dd-a24b-06be952004fc-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:59.430108 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.430000    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:59.430108 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.430014    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:59.430108 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.430027    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/124b9d97-7aec-44dd-a24b-06be952004fc-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:59.772454 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.772418    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd" event={"ID":"dae611da-8694-4348-9dff-90bd553e95ec","Type":"ContainerStarted","Data":"a942d64662474c14cd716d120f3985690aec6ec7b5411be3468afc15aa06877e"}
Nov 02 02:02:59.772944 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.772503    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd" event={"ID":"dae611da-8694-4348-9dff-90bd553e95ec","Type":"ContainerStarted","Data":"1d5fe982ca290531d79bb367087d96c8dbddbab48b2dafd4e6720f55f83ff38b"}
Nov 02 02:02:59.774291 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.774256    5639 generic.go:356] "Generic (PLEG): container finished" podID="124b9d97-7aec-44dd-a24b-06be952004fc" containerID="f35931641094e1695c3108bb58b7c40445d8c2ffafa56c4180fe6d6280b14bd7" exitCode=0
Nov 02 02:02:59.774421 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.774310    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"
Nov 02 02:02:59.774421 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.774306    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd" event={"ID":"124b9d97-7aec-44dd-a24b-06be952004fc","Type":"ContainerDied","Data":"f35931641094e1695c3108bb58b7c40445d8c2ffafa56c4180fe6d6280b14bd7"}
Nov 02 02:02:59.774500 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.774442    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd" event={"ID":"124b9d97-7aec-44dd-a24b-06be952004fc","Type":"ContainerDied","Data":"e2e70ca3ab11f53147a9cd6913782c8a2ce0a3cde20b64fa9952a3987ac7fade"}
Nov 02 02:02:59.774500 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.774464    5639 scope.go:122] "RemoveContainer" containerID="f35931641094e1695c3108bb58b7c40445d8c2ffafa56c4180fe6d6280b14bd7"
Nov 02 02:02:59.786071 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.786055    5639 scope.go:122] "RemoveContainer" containerID="a599f47014a914bad2b2f716f29a0bf30b66f348c4890665f5f2ace86edea778"
Nov 02 02:02:59.795481 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.795462    5639 scope.go:122] "RemoveContainer" containerID="f35931641094e1695c3108bb58b7c40445d8c2ffafa56c4180fe6d6280b14bd7"
Nov 02 02:02:59.795759 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:02:59.795744    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f35931641094e1695c3108bb58b7c40445d8c2ffafa56c4180fe6d6280b14bd7\": container with ID starting with f35931641094e1695c3108bb58b7c40445d8c2ffafa56c4180fe6d6280b14bd7 not found: ID does not exist" containerID="f35931641094e1695c3108bb58b7c40445d8c2ffafa56c4180fe6d6280b14bd7"
Nov 02 02:02:59.795810 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.795767    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f35931641094e1695c3108bb58b7c40445d8c2ffafa56c4180fe6d6280b14bd7"} err="failed to get container status \"f35931641094e1695c3108bb58b7c40445d8c2ffafa56c4180fe6d6280b14bd7\": rpc error: code = NotFound desc = could not find container \"f35931641094e1695c3108bb58b7c40445d8c2ffafa56c4180fe6d6280b14bd7\": container with ID starting with f35931641094e1695c3108bb58b7c40445d8c2ffafa56c4180fe6d6280b14bd7 not found: ID does not exist"
Nov 02 02:02:59.795810 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.795783    5639 scope.go:122] "RemoveContainer" containerID="a599f47014a914bad2b2f716f29a0bf30b66f348c4890665f5f2ace86edea778"
Nov 02 02:02:59.796024 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:02:59.796007    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a599f47014a914bad2b2f716f29a0bf30b66f348c4890665f5f2ace86edea778\": container with ID starting with a599f47014a914bad2b2f716f29a0bf30b66f348c4890665f5f2ace86edea778 not found: ID does not exist" containerID="a599f47014a914bad2b2f716f29a0bf30b66f348c4890665f5f2ace86edea778"
Nov 02 02:02:59.796070 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.796028    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a599f47014a914bad2b2f716f29a0bf30b66f348c4890665f5f2ace86edea778"} err="failed to get container status \"a599f47014a914bad2b2f716f29a0bf30b66f348c4890665f5f2ace86edea778\": rpc error: code = NotFound desc = could not find container \"a599f47014a914bad2b2f716f29a0bf30b66f348c4890665f5f2ace86edea778\": container with ID starting with a599f47014a914bad2b2f716f29a0bf30b66f348c4890665f5f2ace86edea778 not found: ID does not exist"
Nov 02 02:02:59.893553 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:02:59.893505    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd" podStartSLOduration=1.8934917740000001 podStartE2EDuration="1.893491774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:02:59.891309601 +0000 UTC m=+2244.794991714" watchObservedRunningTime="2026-11-02 02:02:59.893491774 +0000 UTC m=+2244.797173887"
Nov 02 02:03:00.032260 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:00.031018    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"]
Nov 02 02:03:00.032260 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:00.031086    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd"]
Nov 02 02:03:00.235909 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:00.235356    5639 patch_prober.go:29] interesting pod/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:03:00.235909 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:00.235426    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-bb0464-openshift-default-6575459fcc-rskzd" podUID="124b9d97-7aec-44dd-a24b-06be952004fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:03:00.497499 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:00.497471    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:03:00.503414 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:00.503387    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:03:00.783254 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:00.783175    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:03:00.784255 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:00.784240    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:03:01.218945 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:01.218908    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="124b9d97-7aec-44dd-a24b-06be952004fc" path="/var/lib/kubelet/pods/124b9d97-7aec-44dd-a24b-06be952004fc/volumes"
Nov 02 02:03:02.044333 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.044249    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 02:03:02.171032 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.170995    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-data\") pod \"5d8da016-d523-4efd-91de-aae563f7f1a6\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") "
Nov 02 02:03:02.171219 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.171047    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-podinfo\") pod \"5d8da016-d523-4efd-91de-aae563f7f1a6\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") "
Nov 02 02:03:02.171219 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.171103    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d8da016-d523-4efd-91de-aae563f7f1a6-kube-api-access-9x8p8\" (UniqueName: \"kubernetes.io/projected/5d8da016-d523-4efd-91de-aae563f7f1a6-kube-api-access-9x8p8\") pod \"5d8da016-d523-4efd-91de-aae563f7f1a6\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") "
Nov 02 02:03:02.171219 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.171153    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-workload-certs\") pod \"5d8da016-d523-4efd-91de-aae563f7f1a6\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") "
Nov 02 02:03:02.171219 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.171204    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d8da016-d523-4efd-91de-aae563f7f1a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8da016-d523-4efd-91de-aae563f7f1a6-istiod-ca-cert\") pod \"5d8da016-d523-4efd-91de-aae563f7f1a6\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") "
Nov 02 02:03:02.171432 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.171270    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-credential-socket\") pod \"5d8da016-d523-4efd-91de-aae563f7f1a6\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") "
Nov 02 02:03:02.171432 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.171306    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-data" pod "5d8da016-d523-4efd-91de-aae563f7f1a6" (UID: "5d8da016-d523-4efd-91de-aae563f7f1a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:02.171432 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.171337    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-token\" (UniqueName: \"kubernetes.io/projected/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-token\") pod \"5d8da016-d523-4efd-91de-aae563f7f1a6\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") "
Nov 02 02:03:02.171432 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.171384    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-workload-socket\") pod \"5d8da016-d523-4efd-91de-aae563f7f1a6\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") "
Nov 02 02:03:02.171631 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.171544    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-envoy\") pod \"5d8da016-d523-4efd-91de-aae563f7f1a6\" (UID: \"5d8da016-d523-4efd-91de-aae563f7f1a6\") "
Nov 02 02:03:02.171756 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.171732    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d8da016-d523-4efd-91de-aae563f7f1a6-istiod-ca-cert" pod "5d8da016-d523-4efd-91de-aae563f7f1a6" (UID: "5d8da016-d523-4efd-91de-aae563f7f1a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:03:02.171881 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.171857    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-workload-socket" pod "5d8da016-d523-4efd-91de-aae563f7f1a6" (UID: "5d8da016-d523-4efd-91de-aae563f7f1a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:02.172007 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.171957    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-credential-socket" pod "5d8da016-d523-4efd-91de-aae563f7f1a6" (UID: "5d8da016-d523-4efd-91de-aae563f7f1a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:02.172084 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.172026    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-workload-certs" pod "5d8da016-d523-4efd-91de-aae563f7f1a6" (UID: "5d8da016-d523-4efd-91de-aae563f7f1a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:02.172487 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.172413    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:02.172487 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.172440    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8da016-d523-4efd-91de-aae563f7f1a6-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:02.172487 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.172453    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:02.172487 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.172467    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:02.172487 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.172478    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:02.173601 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.173576    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d8da016-d523-4efd-91de-aae563f7f1a6-kube-api-access-9x8p8" pod "5d8da016-d523-4efd-91de-aae563f7f1a6" (UID: "5d8da016-d523-4efd-91de-aae563f7f1a6"). InnerVolumeSpecName "kube-api-access-9x8p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:02.173710 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.173636    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-podinfo" pod "5d8da016-d523-4efd-91de-aae563f7f1a6" (UID: "5d8da016-d523-4efd-91de-aae563f7f1a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:03:02.173826 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.173807    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-envoy" pod "5d8da016-d523-4efd-91de-aae563f7f1a6" (UID: "5d8da016-d523-4efd-91de-aae563f7f1a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:02.173960 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.173944    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-token" pod "5d8da016-d523-4efd-91de-aae563f7f1a6" (UID: "5d8da016-d523-4efd-91de-aae563f7f1a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:02.274481 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.274438    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x8p8\" (UniqueName: \"kubernetes.io/projected/5d8da016-d523-4efd-91de-aae563f7f1a6-kube-api-access-9x8p8\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:02.274481 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.274483    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:02.274687 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.274499    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:02.274687 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.274513    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8da016-d523-4efd-91de-aae563f7f1a6-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:02.799023 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.798986    5639 generic.go:356] "Generic (PLEG): container finished" podID="5d8da016-d523-4efd-91de-aae563f7f1a6" containerID="4bbc475bcfb487be3b93e77093d30391682727e237959b009b40910ad2bd6f4a" exitCode=0
Nov 02 02:03:02.799321 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.799046    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"
Nov 02 02:03:02.799321 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.799069    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx" event={"ID":"5d8da016-d523-4efd-91de-aae563f7f1a6","Type":"ContainerDied","Data":"4bbc475bcfb487be3b93e77093d30391682727e237959b009b40910ad2bd6f4a"}
Nov 02 02:03:02.799321 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.799161    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx" event={"ID":"5d8da016-d523-4efd-91de-aae563f7f1a6","Type":"ContainerDied","Data":"71677e740232da8a6b130ca366fee1c45352d8954873124c6f43745a4deaf066"}
Nov 02 02:03:02.799321 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.799188    5639 scope.go:122] "RemoveContainer" containerID="4bbc475bcfb487be3b93e77093d30391682727e237959b009b40910ad2bd6f4a"
Nov 02 02:03:02.813141 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.813117    5639 scope.go:122] "RemoveContainer" containerID="325a54703dad3b6d5437e231b5ef8bcdc8902b51ca8435ee5ee39c9f432b6603"
Nov 02 02:03:02.824016 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.823995    5639 scope.go:122] "RemoveContainer" containerID="4bbc475bcfb487be3b93e77093d30391682727e237959b009b40910ad2bd6f4a"
Nov 02 02:03:02.824406 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:03:02.824381    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bbc475bcfb487be3b93e77093d30391682727e237959b009b40910ad2bd6f4a\": container with ID starting with 4bbc475bcfb487be3b93e77093d30391682727e237959b009b40910ad2bd6f4a not found: ID does not exist" containerID="4bbc475bcfb487be3b93e77093d30391682727e237959b009b40910ad2bd6f4a"
Nov 02 02:03:02.824472 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.824417    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bbc475bcfb487be3b93e77093d30391682727e237959b009b40910ad2bd6f4a"} err="failed to get container status \"4bbc475bcfb487be3b93e77093d30391682727e237959b009b40910ad2bd6f4a\": rpc error: code = NotFound desc = could not find container \"4bbc475bcfb487be3b93e77093d30391682727e237959b009b40910ad2bd6f4a\": container with ID starting with 4bbc475bcfb487be3b93e77093d30391682727e237959b009b40910ad2bd6f4a not found: ID does not exist"
Nov 02 02:03:02.824472 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.824447    5639 scope.go:122] "RemoveContainer" containerID="325a54703dad3b6d5437e231b5ef8bcdc8902b51ca8435ee5ee39c9f432b6603"
Nov 02 02:03:02.824709 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:03:02.824695    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"325a54703dad3b6d5437e231b5ef8bcdc8902b51ca8435ee5ee39c9f432b6603\": container with ID starting with 325a54703dad3b6d5437e231b5ef8bcdc8902b51ca8435ee5ee39c9f432b6603 not found: ID does not exist" containerID="325a54703dad3b6d5437e231b5ef8bcdc8902b51ca8435ee5ee39c9f432b6603"
Nov 02 02:03:02.824754 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.824712    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325a54703dad3b6d5437e231b5ef8bcdc8902b51ca8435ee5ee39c9f432b6603"} err="failed to get container status \"325a54703dad3b6d5437e231b5ef8bcdc8902b51ca8435ee5ee39c9f432b6603\": rpc error: code = NotFound desc = could not find container \"325a54703dad3b6d5437e231b5ef8bcdc8902b51ca8435ee5ee39c9f432b6603\": container with ID starting with 325a54703dad3b6d5437e231b5ef8bcdc8902b51ca8435ee5ee39c9f432b6603 not found: ID does not exist"
Nov 02 02:03:02.975913 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:02.975323    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"]
Nov 02 02:03:03.034304 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:03.033812    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-19436b-openshift-default-84474f7c47-t9spx"]
Nov 02 02:03:03.238307 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:03.237857    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d8da016-d523-4efd-91de-aae563f7f1a6" path="/var/lib/kubelet/pods/5d8da016-d523-4efd-91de-aae563f7f1a6/volumes"
Nov 02 02:03:06.702303 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.692790    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4"]
Nov 02 02:03:06.702303 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.694058    5639 cpu_manager.go:414] "containerMap: removing container" podUID="124b9d97-7aec-44dd-a24b-06be952004fc" containerName="istio-proxy"
Nov 02 02:03:06.702303 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.694088    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="124b9d97-7aec-44dd-a24b-06be952004fc" containerName="istio-proxy"
Nov 02 02:03:06.702303 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.694110    5639 cpu_manager.go:414] "containerMap: removing container" podUID="5d8da016-d523-4efd-91de-aae563f7f1a6" containerName="istio-proxy"
Nov 02 02:03:06.702303 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.694117    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d8da016-d523-4efd-91de-aae563f7f1a6" containerName="istio-proxy"
Nov 02 02:03:06.702303 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.694414    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d8da016-d523-4efd-91de-aae563f7f1a6" containerName="istio-proxy"
Nov 02 02:03:06.702303 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.694445    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="124b9d97-7aec-44dd-a24b-06be952004fc" containerName="istio-proxy"
Nov 02 02:03:06.702303 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.694459    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="124b9d97-7aec-44dd-a24b-06be952004fc" containerName="istio-proxy"
Nov 02 02:03:06.726339 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.725119    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4"
Nov 02 02:03:06.733306 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.729425    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4"]
Nov 02 02:03:06.753877 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.749662    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91fe93cf-734d-4a19-868d-b4c38a76bc2b-plugins-config-volume\") pod \"payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4\" (UID: \"91fe93cf-734d-4a19-868d-b4c38a76bc2b\") " pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4"
Nov 02 02:03:06.753877 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.749781    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2bjc\" (UniqueName: \"kubernetes.io/projected/91fe93cf-734d-4a19-868d-b4c38a76bc2b-kube-api-access-l2bjc\") pod \"payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4\" (UID: \"91fe93cf-734d-4a19-868d-b4c38a76bc2b\") " pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4"
Nov 02 02:03:06.827534 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.827505    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-07ac70d0-dockercfg-smbpl\""
Nov 02 02:03:06.827534 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.827515    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-07ac70d0\""
Nov 02 02:03:06.832989 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.832960    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf"]
Nov 02 02:03:06.833924 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.833895    5639 cpu_manager.go:414] "containerMap: removing container" podUID="124b9d97-7aec-44dd-a24b-06be952004fc" containerName="istio-proxy"
Nov 02 02:03:06.833924 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.833917    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="124b9d97-7aec-44dd-a24b-06be952004fc" containerName="istio-proxy"
Nov 02 02:03:06.834112 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.833932    5639 cpu_manager.go:414] "containerMap: removing container" podUID="5d8da016-d523-4efd-91de-aae563f7f1a6" containerName="istio-proxy"
Nov 02 02:03:06.834112 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.833938    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d8da016-d523-4efd-91de-aae563f7f1a6" containerName="istio-proxy"
Nov 02 02:03:06.834112 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.834069    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d8da016-d523-4efd-91de-aae563f7f1a6" containerName="istio-proxy"
Nov 02 02:03:06.837603 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.837583    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf"
Nov 02 02:03:06.850966 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.850942    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2bjc\" (UniqueName: \"kubernetes.io/projected/91fe93cf-734d-4a19-868d-b4c38a76bc2b-kube-api-access-l2bjc\") pod \"payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4\" (UID: \"91fe93cf-734d-4a19-868d-b4c38a76bc2b\") " pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4"
Nov 02 02:03:06.851116 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.851033    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/322c165c-ab83-4cc0-91f2-d83533aa6472-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf\" (UID: \"322c165c-ab83-4cc0-91f2-d83533aa6472\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf"
Nov 02 02:03:06.851233 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.851209    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk2sm\" (UniqueName: \"kubernetes.io/projected/322c165c-ab83-4cc0-91f2-d83533aa6472-kube-api-access-gk2sm\") pod \"payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf\" (UID: \"322c165c-ab83-4cc0-91f2-d83533aa6472\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf"
Nov 02 02:03:06.851300 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.851274    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91fe93cf-734d-4a19-868d-b4c38a76bc2b-plugins-config-volume\") pod \"payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4\" (UID: \"91fe93cf-734d-4a19-868d-b4c38a76bc2b\") " pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4"
Nov 02 02:03:06.851860 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.851838    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91fe93cf-734d-4a19-868d-b4c38a76bc2b-plugins-config-volume\") pod \"payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4\" (UID: \"91fe93cf-734d-4a19-868d-b4c38a76bc2b\") " pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4"
Nov 02 02:03:06.894886 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.894857    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf"]
Nov 02 02:03:06.912052 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.912017    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2bjc\" (UniqueName: \"kubernetes.io/projected/91fe93cf-734d-4a19-868d-b4c38a76bc2b-kube-api-access-l2bjc\") pod \"payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4\" (UID: \"91fe93cf-734d-4a19-868d-b4c38a76bc2b\") " pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4"
Nov 02 02:03:06.952891 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.952619    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/322c165c-ab83-4cc0-91f2-d83533aa6472-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf\" (UID: \"322c165c-ab83-4cc0-91f2-d83533aa6472\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf"
Nov 02 02:03:06.953066 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.952930    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk2sm\" (UniqueName: \"kubernetes.io/projected/322c165c-ab83-4cc0-91f2-d83533aa6472-kube-api-access-gk2sm\") pod \"payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf\" (UID: \"322c165c-ab83-4cc0-91f2-d83533aa6472\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf"
Nov 02 02:03:06.953590 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:06.953572    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/322c165c-ab83-4cc0-91f2-d83533aa6472-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf\" (UID: \"322c165c-ab83-4cc0-91f2-d83533aa6472\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf"
Nov 02 02:03:07.013787 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:07.013758    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk2sm\" (UniqueName: \"kubernetes.io/projected/322c165c-ab83-4cc0-91f2-d83533aa6472-kube-api-access-gk2sm\") pod \"payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf\" (UID: \"322c165c-ab83-4cc0-91f2-d83533aa6472\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf"
Nov 02 02:03:07.069524 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:07.069493    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4"
Nov 02 02:03:07.154700 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:07.154655    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf"
Nov 02 02:03:07.334266 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:07.334227    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-187478-564b56656f-xfg79"]
Nov 02 02:03:07.434245 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:07.434210    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j"]
Nov 02 02:03:07.450600 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:07.450573    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4"]
Nov 02 02:03:07.451223 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:03:07.451200    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91fe93cf_734d_4a19_868d_b4c38a76bc2b.slice/crio-69c93929ee085c681833d9d480106a40fbbd090a5ec4692cdd10877e5e309748 WatchSource:0}: Error finding container 69c93929ee085c681833d9d480106a40fbbd090a5ec4692cdd10877e5e309748: Status 404 returned error can't find the container with id 69c93929ee085c681833d9d480106a40fbbd090a5ec4692cdd10877e5e309748
Nov 02 02:03:07.564492 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:07.564459    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf"]
Nov 02 02:03:07.565613 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:03:07.565588    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322c165c_ab83_4cc0_91f2_d83533aa6472.slice/crio-321b1de947a23be8195463a174bfe8795d8abcda8a80328b6cf98b68943a666a WatchSource:0}: Error finding container 321b1de947a23be8195463a174bfe8795d8abcda8a80328b6cf98b68943a666a: Status 404 returned error can't find the container with id 321b1de947a23be8195463a174bfe8795d8abcda8a80328b6cf98b68943a666a
Nov 02 02:03:07.840102 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:07.840067    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf" event={"ID":"322c165c-ab83-4cc0-91f2-d83533aa6472","Type":"ContainerStarted","Data":"f6ea5b2fbb2ce852bb231bbb9131c2b5b3c248c91a8d9f6b70fdb9da31e99389"}
Nov 02 02:03:07.840643 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:07.840137    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf"
Nov 02 02:03:07.840643 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:07.840163    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf" event={"ID":"322c165c-ab83-4cc0-91f2-d83533aa6472","Type":"ContainerStarted","Data":"321b1de947a23be8195463a174bfe8795d8abcda8a80328b6cf98b68943a666a"}
Nov 02 02:03:07.841806 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:07.841781    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4" event={"ID":"91fe93cf-734d-4a19-868d-b4c38a76bc2b","Type":"ContainerStarted","Data":"20ad0b7f82df36994903072d915deaef626076d57ad891ac85882d0b6eecebce"}
Nov 02 02:03:07.841903 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:07.841826    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4" event={"ID":"91fe93cf-734d-4a19-868d-b4c38a76bc2b","Type":"ContainerStarted","Data":"69c93929ee085c681833d9d480106a40fbbd090a5ec4692cdd10877e5e309748"}
Nov 02 02:03:07.841903 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:07.841882    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4"
Nov 02 02:03:07.957201 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:07.957137    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf" podStartSLOduration=1.957118581 podStartE2EDuration="1.957118581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:03:07.955393408 +0000 UTC m=+2252.859075528" watchObservedRunningTime="2026-11-02 02:03:07.957118581 +0000 UTC m=+2252.860800696"
Nov 02 02:03:12.335454 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.335413    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-187478-564b56656f-xfg79" podUID="67dc6edc-42f7-4e61-ad3c-38d3c3be4049" containerName="payload-processing" containerID="cri-o://f7aa833139a2978502ef2d38fba0f4721bb2c8f1d3447b716a82de85e122b721" gracePeriod=25
Nov 02 02:03:12.435199 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.435150    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j" podUID="ef4d980a-8ce2-4357-aadc-d1839d056a61" containerName="payload-pre-processing" containerID="cri-o://42927e6f4c4a09dab66ce743c20ef756601aafcfaab15329876057edb9ac9fc0" gracePeriod=25
Nov 02 02:03:12.620623 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.620591    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-187478-564b56656f-xfg79"
Nov 02 02:03:12.679407 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.679372    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67dc6edc-42f7-4e61-ad3c-38d3c3be4049-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67dc6edc-42f7-4e61-ad3c-38d3c3be4049-plugins-config-volume\") pod \"67dc6edc-42f7-4e61-ad3c-38d3c3be4049\" (UID: \"67dc6edc-42f7-4e61-ad3c-38d3c3be4049\") "
Nov 02 02:03:12.679587 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.679516    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67dc6edc-42f7-4e61-ad3c-38d3c3be4049-kube-api-access-vkt67\" (UniqueName: \"kubernetes.io/projected/67dc6edc-42f7-4e61-ad3c-38d3c3be4049-kube-api-access-vkt67\") pod \"67dc6edc-42f7-4e61-ad3c-38d3c3be4049\" (UID: \"67dc6edc-42f7-4e61-ad3c-38d3c3be4049\") "
Nov 02 02:03:12.679827 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.679798    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67dc6edc-42f7-4e61-ad3c-38d3c3be4049-plugins-config-volume" pod "67dc6edc-42f7-4e61-ad3c-38d3c3be4049" (UID: "67dc6edc-42f7-4e61-ad3c-38d3c3be4049"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:03:12.680067 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.680042    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67dc6edc-42f7-4e61-ad3c-38d3c3be4049-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:12.682750 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.682716    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67dc6edc-42f7-4e61-ad3c-38d3c3be4049-kube-api-access-vkt67" pod "67dc6edc-42f7-4e61-ad3c-38d3c3be4049" (UID: "67dc6edc-42f7-4e61-ad3c-38d3c3be4049"). InnerVolumeSpecName "kube-api-access-vkt67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:12.710219 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.710199    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j"
Nov 02 02:03:12.776006 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.775945    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4" podStartSLOduration=6.775927414 podStartE2EDuration="6.775927414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:03:08.082737557 +0000 UTC m=+2252.986419672" watchObservedRunningTime="2026-11-02 02:03:12.775927414 +0000 UTC m=+2257.679609529"
Nov 02 02:03:12.780629 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.780599    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef4d980a-8ce2-4357-aadc-d1839d056a61-kube-api-access-hj7t7\" (UniqueName: \"kubernetes.io/projected/ef4d980a-8ce2-4357-aadc-d1839d056a61-kube-api-access-hj7t7\") pod \"ef4d980a-8ce2-4357-aadc-d1839d056a61\" (UID: \"ef4d980a-8ce2-4357-aadc-d1839d056a61\") "
Nov 02 02:03:12.780777 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.780693    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef4d980a-8ce2-4357-aadc-d1839d056a61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef4d980a-8ce2-4357-aadc-d1839d056a61-plugins-config-volume\") pod \"ef4d980a-8ce2-4357-aadc-d1839d056a61\" (UID: \"ef4d980a-8ce2-4357-aadc-d1839d056a61\") "
Nov 02 02:03:12.781147 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.781115    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef4d980a-8ce2-4357-aadc-d1839d056a61-plugins-config-volume" pod "ef4d980a-8ce2-4357-aadc-d1839d056a61" (UID: "ef4d980a-8ce2-4357-aadc-d1839d056a61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:03:12.781438 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.781373    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkt67\" (UniqueName: \"kubernetes.io/projected/67dc6edc-42f7-4e61-ad3c-38d3c3be4049-kube-api-access-vkt67\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:12.781438 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.781396    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef4d980a-8ce2-4357-aadc-d1839d056a61-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:12.783033 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.783008    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef4d980a-8ce2-4357-aadc-d1839d056a61-kube-api-access-hj7t7" pod "ef4d980a-8ce2-4357-aadc-d1839d056a61" (UID: "ef4d980a-8ce2-4357-aadc-d1839d056a61"). InnerVolumeSpecName "kube-api-access-hj7t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:12.882902 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.882822    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj7t7\" (UniqueName: \"kubernetes.io/projected/ef4d980a-8ce2-4357-aadc-d1839d056a61-kube-api-access-hj7t7\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:12.885784 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.885758    5639 generic.go:356] "Generic (PLEG): container finished" podID="ef4d980a-8ce2-4357-aadc-d1839d056a61" containerID="42927e6f4c4a09dab66ce743c20ef756601aafcfaab15329876057edb9ac9fc0" exitCode=0
Nov 02 02:03:12.885930 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.885821    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j"
Nov 02 02:03:12.885930 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.885848    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j" event={"ID":"ef4d980a-8ce2-4357-aadc-d1839d056a61","Type":"ContainerDied","Data":"42927e6f4c4a09dab66ce743c20ef756601aafcfaab15329876057edb9ac9fc0"}
Nov 02 02:03:12.885930 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.885928    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j" event={"ID":"ef4d980a-8ce2-4357-aadc-d1839d056a61","Type":"ContainerDied","Data":"57c252698c3e1ff3517cdd54f22d0d7cc9a4dd4fa0f607f0bbbbbd5acb22e5c5"}
Nov 02 02:03:12.886102 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.885952    5639 scope.go:122] "RemoveContainer" containerID="42927e6f4c4a09dab66ce743c20ef756601aafcfaab15329876057edb9ac9fc0"
Nov 02 02:03:12.887458 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.887435    5639 generic.go:356] "Generic (PLEG): container finished" podID="67dc6edc-42f7-4e61-ad3c-38d3c3be4049" containerID="f7aa833139a2978502ef2d38fba0f4721bb2c8f1d3447b716a82de85e122b721" exitCode=0
Nov 02 02:03:12.887588 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.887474    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-187478-564b56656f-xfg79"
Nov 02 02:03:12.887588 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.887526    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-187478-564b56656f-xfg79" event={"ID":"67dc6edc-42f7-4e61-ad3c-38d3c3be4049","Type":"ContainerDied","Data":"f7aa833139a2978502ef2d38fba0f4721bb2c8f1d3447b716a82de85e122b721"}
Nov 02 02:03:12.887693 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.887617    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-187478-564b56656f-xfg79" event={"ID":"67dc6edc-42f7-4e61-ad3c-38d3c3be4049","Type":"ContainerDied","Data":"f63d875de1514949074e77e1215dbb9ffe19243c3705e41504e7544a0ed598b5"}
Nov 02 02:03:12.901664 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.901637    5639 scope.go:122] "RemoveContainer" containerID="42927e6f4c4a09dab66ce743c20ef756601aafcfaab15329876057edb9ac9fc0"
Nov 02 02:03:12.902084 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:03:12.902008    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42927e6f4c4a09dab66ce743c20ef756601aafcfaab15329876057edb9ac9fc0\": container with ID starting with 42927e6f4c4a09dab66ce743c20ef756601aafcfaab15329876057edb9ac9fc0 not found: ID does not exist" containerID="42927e6f4c4a09dab66ce743c20ef756601aafcfaab15329876057edb9ac9fc0"
Nov 02 02:03:12.902084 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.902053    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42927e6f4c4a09dab66ce743c20ef756601aafcfaab15329876057edb9ac9fc0"} err="failed to get container status \"42927e6f4c4a09dab66ce743c20ef756601aafcfaab15329876057edb9ac9fc0\": rpc error: code = NotFound desc = could not find container \"42927e6f4c4a09dab66ce743c20ef756601aafcfaab15329876057edb9ac9fc0\": container with ID starting with 42927e6f4c4a09dab66ce743c20ef756601aafcfaab15329876057edb9ac9fc0 not found: ID does not exist"
Nov 02 02:03:12.902084 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.902072    5639 scope.go:122] "RemoveContainer" containerID="f7aa833139a2978502ef2d38fba0f4721bb2c8f1d3447b716a82de85e122b721"
Nov 02 02:03:12.915274 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.915249    5639 scope.go:122] "RemoveContainer" containerID="f7aa833139a2978502ef2d38fba0f4721bb2c8f1d3447b716a82de85e122b721"
Nov 02 02:03:12.915616 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:03:12.915588    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7aa833139a2978502ef2d38fba0f4721bb2c8f1d3447b716a82de85e122b721\": container with ID starting with f7aa833139a2978502ef2d38fba0f4721bb2c8f1d3447b716a82de85e122b721 not found: ID does not exist" containerID="f7aa833139a2978502ef2d38fba0f4721bb2c8f1d3447b716a82de85e122b721"
Nov 02 02:03:12.915708 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:12.915625    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7aa833139a2978502ef2d38fba0f4721bb2c8f1d3447b716a82de85e122b721"} err="failed to get container status \"f7aa833139a2978502ef2d38fba0f4721bb2c8f1d3447b716a82de85e122b721\": rpc error: code = NotFound desc = could not find container \"f7aa833139a2978502ef2d38fba0f4721bb2c8f1d3447b716a82de85e122b721\": container with ID starting with f7aa833139a2978502ef2d38fba0f4721bb2c8f1d3447b716a82de85e122b721 not found: ID does not exist"
Nov 02 02:03:13.108338 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:13.108303    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-187478-564b56656f-xfg79"]
Nov 02 02:03:13.112526 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:13.112500    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-187478-564b56656f-xfg79"]
Nov 02 02:03:13.219667 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:13.219585    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67dc6edc-42f7-4e61-ad3c-38d3c3be4049" path="/var/lib/kubelet/pods/67dc6edc-42f7-4e61-ad3c-38d3c3be4049/volumes"
Nov 02 02:03:13.221595 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:13.221569    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j"]
Nov 02 02:03:13.225261 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:13.225237    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-187478-5bfdbb8479-pm57j"]
Nov 02 02:03:14.440245 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.440203    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main" containerID="cri-o://2dccefb602dc3bde210391df8e1a13c0d9d5c0a5bdc36c2bff8b7131e0d98edc" gracePeriod=45
Nov 02 02:03:14.703220 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.703155    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 02:03:14.805740 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.805709    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-tmp-dir\") pod \"7d97942c-a6d9-4272-ac79-51a0986061b7\" (UID: \"7d97942c-a6d9-4272-ac79-51a0986061b7\") "
Nov 02 02:03:14.805924 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.805820    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-home\" (UniqueName: \"kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-home\") pod \"7d97942c-a6d9-4272-ac79-51a0986061b7\" (UID: \"7d97942c-a6d9-4272-ac79-51a0986061b7\") "
Nov 02 02:03:14.806005 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.805920    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d97942c-a6d9-4272-ac79-51a0986061b7-kube-api-access-ftjxl\" (UniqueName: \"kubernetes.io/projected/7d97942c-a6d9-4272-ac79-51a0986061b7-kube-api-access-ftjxl\") pod \"7d97942c-a6d9-4272-ac79-51a0986061b7\" (UID: \"7d97942c-a6d9-4272-ac79-51a0986061b7\") "
Nov 02 02:03:14.806005 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.805959    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-model-cache\") pod \"7d97942c-a6d9-4272-ac79-51a0986061b7\" (UID: \"7d97942c-a6d9-4272-ac79-51a0986061b7\") "
Nov 02 02:03:14.806130 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.806001    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-tmp-dir" pod "7d97942c-a6d9-4272-ac79-51a0986061b7" (UID: "7d97942c-a6d9-4272-ac79-51a0986061b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:14.806130 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.806047    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d97942c-a6d9-4272-ac79-51a0986061b7-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d97942c-a6d9-4272-ac79-51a0986061b7-tls-certs\") pod \"7d97942c-a6d9-4272-ac79-51a0986061b7\" (UID: \"7d97942c-a6d9-4272-ac79-51a0986061b7\") "
Nov 02 02:03:14.806130 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.806113    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-dshm\") pod \"7d97942c-a6d9-4272-ac79-51a0986061b7\" (UID: \"7d97942c-a6d9-4272-ac79-51a0986061b7\") "
Nov 02 02:03:14.806321 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.806136    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-home" pod "7d97942c-a6d9-4272-ac79-51a0986061b7" (UID: "7d97942c-a6d9-4272-ac79-51a0986061b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:14.806321 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.806238    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-model-cache" pod "7d97942c-a6d9-4272-ac79-51a0986061b7" (UID: "7d97942c-a6d9-4272-ac79-51a0986061b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:14.806883 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.806859    5639 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-tmp-dir\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:14.806978 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.806889    5639 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-home\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:14.806978 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.806903    5639 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-model-cache\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:14.808606 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.808581    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d97942c-a6d9-4272-ac79-51a0986061b7-kube-api-access-ftjxl" pod "7d97942c-a6d9-4272-ac79-51a0986061b7" (UID: "7d97942c-a6d9-4272-ac79-51a0986061b7"). InnerVolumeSpecName "kube-api-access-ftjxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:14.808821 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.808804    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d97942c-a6d9-4272-ac79-51a0986061b7-tls-certs" pod "7d97942c-a6d9-4272-ac79-51a0986061b7" (UID: "7d97942c-a6d9-4272-ac79-51a0986061b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:03:14.811602 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.811577    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-dshm" pod "7d97942c-a6d9-4272-ac79-51a0986061b7" (UID: "7d97942c-a6d9-4272-ac79-51a0986061b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:14.908032 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.907999    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftjxl\" (UniqueName: \"kubernetes.io/projected/7d97942c-a6d9-4272-ac79-51a0986061b7-kube-api-access-ftjxl\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:14.908032 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.908034    5639 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d97942c-a6d9-4272-ac79-51a0986061b7-tls-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:14.908322 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.908049    5639 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d97942c-a6d9-4272-ac79-51a0986061b7-dshm\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:14.908650 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.908610    5639 generic.go:356] "Generic (PLEG): container finished" podID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerID="2dccefb602dc3bde210391df8e1a13c0d9d5c0a5bdc36c2bff8b7131e0d98edc" exitCode=0
Nov 02 02:03:14.908785 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.908688    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"
Nov 02 02:03:14.908861 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.908812    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" event={"ID":"7d97942c-a6d9-4272-ac79-51a0986061b7","Type":"ContainerDied","Data":"2dccefb602dc3bde210391df8e1a13c0d9d5c0a5bdc36c2bff8b7131e0d98edc"}
Nov 02 02:03:14.908861 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.908845    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c" event={"ID":"7d97942c-a6d9-4272-ac79-51a0986061b7","Type":"ContainerDied","Data":"c1b4d976aaf83d14771b3810f56015689daacad54928b3ee7775867963d94eea"}
Nov 02 02:03:14.908980 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.908866    5639 scope.go:122] "RemoveContainer" containerID="2dccefb602dc3bde210391df8e1a13c0d9d5c0a5bdc36c2bff8b7131e0d98edc"
Nov 02 02:03:14.921984 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.921963    5639 scope.go:122] "RemoveContainer" containerID="2dccefb602dc3bde210391df8e1a13c0d9d5c0a5bdc36c2bff8b7131e0d98edc"
Nov 02 02:03:14.922362 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:03:14.922340    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dccefb602dc3bde210391df8e1a13c0d9d5c0a5bdc36c2bff8b7131e0d98edc\": container with ID starting with 2dccefb602dc3bde210391df8e1a13c0d9d5c0a5bdc36c2bff8b7131e0d98edc not found: ID does not exist" containerID="2dccefb602dc3bde210391df8e1a13c0d9d5c0a5bdc36c2bff8b7131e0d98edc"
Nov 02 02:03:14.922468 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:14.922369    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dccefb602dc3bde210391df8e1a13c0d9d5c0a5bdc36c2bff8b7131e0d98edc"} err="failed to get container status \"2dccefb602dc3bde210391df8e1a13c0d9d5c0a5bdc36c2bff8b7131e0d98edc\": rpc error: code = NotFound desc = could not find container \"2dccefb602dc3bde210391df8e1a13c0d9d5c0a5bdc36c2bff8b7131e0d98edc\": container with ID starting with 2dccefb602dc3bde210391df8e1a13c0d9d5c0a5bdc36c2bff8b7131e0d98edc not found: ID does not exist"
Nov 02 02:03:15.085034 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.084997    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"]
Nov 02 02:03:15.087842 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.087812    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-187478/simulator-w4-187478-kserve-c4b76d8d-cr66c"]
Nov 02 02:03:15.228896 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.228861    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" path="/var/lib/kubelet/pods/7d97942c-a6d9-4272-ac79-51a0986061b7/volumes"
Nov 02 02:03:15.229308 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.229292    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef4d980a-8ce2-4357-aadc-d1839d056a61" path="/var/lib/kubelet/pods/ef4d980a-8ce2-4357-aadc-d1839d056a61/volumes"
Nov 02 02:03:15.310400 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.310357    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9657df10-g8cdq"]
Nov 02 02:03:15.310974 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.310961    5639 cpu_manager.go:414] "containerMap: removing container" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main"
Nov 02 02:03:15.311020 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.310975    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main"
Nov 02 02:03:15.311020 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.310987    5639 cpu_manager.go:414] "containerMap: removing container" podUID="ef4d980a-8ce2-4357-aadc-d1839d056a61" containerName="payload-pre-processing"
Nov 02 02:03:15.311020 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.310992    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef4d980a-8ce2-4357-aadc-d1839d056a61" containerName="payload-pre-processing"
Nov 02 02:03:15.311020 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.311010    5639 cpu_manager.go:414] "containerMap: removing container" podUID="67dc6edc-42f7-4e61-ad3c-38d3c3be4049" containerName="payload-processing"
Nov 02 02:03:15.311020 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.311014    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67dc6edc-42f7-4e61-ad3c-38d3c3be4049" containerName="payload-processing"
Nov 02 02:03:15.311182 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.311101    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d97942c-a6d9-4272-ac79-51a0986061b7" containerName="main"
Nov 02 02:03:15.311182 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.311108    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef4d980a-8ce2-4357-aadc-d1839d056a61" containerName="payload-pre-processing"
Nov 02 02:03:15.311182 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.311117    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="67dc6edc-42f7-4e61-ad3c-38d3c3be4049" containerName="payload-processing"
Nov 02 02:03:15.315660 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.315641    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657df10-g8cdq"
Nov 02 02:03:15.365731 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.365635    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-537ee113-kwdbr"]
Nov 02 02:03:15.368976 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.368949    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:03:15.369174 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.369109    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:03:15.369399 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.369377    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:03:15.369549 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.369530    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:03:15.369905 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.369887    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-537ee113-kwdbr"
Nov 02 02:03:15.373459 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.373435    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9657df10-g8cdq"]
Nov 02 02:03:15.414218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.414180    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1964d9f-bf5f-444f-a781-3284985a726a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9657df10-g8cdq\" (UID: \"a1964d9f-bf5f-444f-a781-3284985a726a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657df10-g8cdq"
Nov 02 02:03:15.414403 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.414263    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/875bffb8-7c4b-44dc-97fe-3be42c9ffe93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-537ee113-kwdbr\" (UID: \"875bffb8-7c4b-44dc-97fe-3be42c9ffe93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-537ee113-kwdbr"
Nov 02 02:03:15.414403 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.414353    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfpls\" (UniqueName: \"kubernetes.io/projected/875bffb8-7c4b-44dc-97fe-3be42c9ffe93-kube-api-access-pfpls\") pod \"maas-api-revoke-sub-537ee113-kwdbr\" (UID: \"875bffb8-7c4b-44dc-97fe-3be42c9ffe93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-537ee113-kwdbr"
Nov 02 02:03:15.414559 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.414541    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdwf5\" (UniqueName: \"kubernetes.io/projected/a1964d9f-bf5f-444f-a781-3284985a726a-kube-api-access-wdwf5\") pod \"maas-api-revoke-sub-9657df10-g8cdq\" (UID: \"a1964d9f-bf5f-444f-a781-3284985a726a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657df10-g8cdq"
Nov 02 02:03:15.414605 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.414597    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1964d9f-bf5f-444f-a781-3284985a726a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9657df10-g8cdq\" (UID: \"a1964d9f-bf5f-444f-a781-3284985a726a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657df10-g8cdq"
Nov 02 02:03:15.414663 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.414616    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/875bffb8-7c4b-44dc-97fe-3be42c9ffe93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-537ee113-kwdbr\" (UID: \"875bffb8-7c4b-44dc-97fe-3be42c9ffe93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-537ee113-kwdbr"
Nov 02 02:03:15.427351 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.427320    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-537ee113-kwdbr"]
Nov 02 02:03:15.515921 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.515880    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdwf5\" (UniqueName: \"kubernetes.io/projected/a1964d9f-bf5f-444f-a781-3284985a726a-kube-api-access-wdwf5\") pod \"maas-api-revoke-sub-9657df10-g8cdq\" (UID: \"a1964d9f-bf5f-444f-a781-3284985a726a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657df10-g8cdq"
Nov 02 02:03:15.516469 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.515948    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1964d9f-bf5f-444f-a781-3284985a726a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9657df10-g8cdq\" (UID: \"a1964d9f-bf5f-444f-a781-3284985a726a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657df10-g8cdq"
Nov 02 02:03:15.516469 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.515977    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/875bffb8-7c4b-44dc-97fe-3be42c9ffe93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-537ee113-kwdbr\" (UID: \"875bffb8-7c4b-44dc-97fe-3be42c9ffe93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-537ee113-kwdbr"
Nov 02 02:03:15.516469 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.516122    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1964d9f-bf5f-444f-a781-3284985a726a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9657df10-g8cdq\" (UID: \"a1964d9f-bf5f-444f-a781-3284985a726a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657df10-g8cdq"
Nov 02 02:03:15.516469 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.516205    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/875bffb8-7c4b-44dc-97fe-3be42c9ffe93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-537ee113-kwdbr\" (UID: \"875bffb8-7c4b-44dc-97fe-3be42c9ffe93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-537ee113-kwdbr"
Nov 02 02:03:15.516469 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.516237    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfpls\" (UniqueName: \"kubernetes.io/projected/875bffb8-7c4b-44dc-97fe-3be42c9ffe93-kube-api-access-pfpls\") pod \"maas-api-revoke-sub-537ee113-kwdbr\" (UID: \"875bffb8-7c4b-44dc-97fe-3be42c9ffe93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-537ee113-kwdbr"
Nov 02 02:03:15.516798 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.516776    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1964d9f-bf5f-444f-a781-3284985a726a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9657df10-g8cdq\" (UID: \"a1964d9f-bf5f-444f-a781-3284985a726a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657df10-g8cdq"
Nov 02 02:03:15.517013 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.516822    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/875bffb8-7c4b-44dc-97fe-3be42c9ffe93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-537ee113-kwdbr\" (UID: \"875bffb8-7c4b-44dc-97fe-3be42c9ffe93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-537ee113-kwdbr"
Nov 02 02:03:15.518660 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.518637    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/875bffb8-7c4b-44dc-97fe-3be42c9ffe93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-537ee113-kwdbr\" (UID: \"875bffb8-7c4b-44dc-97fe-3be42c9ffe93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-537ee113-kwdbr"
Nov 02 02:03:15.518813 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.518798    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1964d9f-bf5f-444f-a781-3284985a726a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9657df10-g8cdq\" (UID: \"a1964d9f-bf5f-444f-a781-3284985a726a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657df10-g8cdq"
Nov 02 02:03:15.576558 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.576519    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdwf5\" (UniqueName: \"kubernetes.io/projected/a1964d9f-bf5f-444f-a781-3284985a726a-kube-api-access-wdwf5\") pod \"maas-api-revoke-sub-9657df10-g8cdq\" (UID: \"a1964d9f-bf5f-444f-a781-3284985a726a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657df10-g8cdq"
Nov 02 02:03:15.576876 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.576856    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfpls\" (UniqueName: \"kubernetes.io/projected/875bffb8-7c4b-44dc-97fe-3be42c9ffe93-kube-api-access-pfpls\") pod \"maas-api-revoke-sub-537ee113-kwdbr\" (UID: \"875bffb8-7c4b-44dc-97fe-3be42c9ffe93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-537ee113-kwdbr"
Nov 02 02:03:15.629040 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.628973    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657df10-g8cdq"
Nov 02 02:03:15.690605 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:15.690571    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-537ee113-kwdbr"
Nov 02 02:03:16.008258 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:16.008235    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9657df10-g8cdq"]
Nov 02 02:03:16.009420 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:03:16.009390    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1964d9f_bf5f_444f_a781_3284985a726a.slice/crio-5f12f11e7c65ac49138e54f4f0004323c7ebcf9baecb2bea20baa820134dfc1e WatchSource:0}: Error finding container 5f12f11e7c65ac49138e54f4f0004323c7ebcf9baecb2bea20baa820134dfc1e: Status 404 returned error can't find the container with id 5f12f11e7c65ac49138e54f4f0004323c7ebcf9baecb2bea20baa820134dfc1e
Nov 02 02:03:16.109793 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:16.109754    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-537ee113-kwdbr"]
Nov 02 02:03:16.110437 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:03:16.110387    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod875bffb8_7c4b_44dc_97fe_3be42c9ffe93.slice/crio-364e435970843c1dd887d44660ad9a1bac6e4e17aa830373fb1fffff70afe251 WatchSource:0}: Error finding container 364e435970843c1dd887d44660ad9a1bac6e4e17aa830373fb1fffff70afe251: Status 404 returned error can't find the container with id 364e435970843c1dd887d44660ad9a1bac6e4e17aa830373fb1fffff70afe251
Nov 02 02:03:16.930829 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:16.930792    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-537ee113-kwdbr" event={"ID":"875bffb8-7c4b-44dc-97fe-3be42c9ffe93","Type":"ContainerStarted","Data":"d749c0c7b223a006350800493d57af6b3aa98dc8a0aa525db35b41b81297a0da"}
Nov 02 02:03:16.931360 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:16.930872    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-537ee113-kwdbr" event={"ID":"875bffb8-7c4b-44dc-97fe-3be42c9ffe93","Type":"ContainerStarted","Data":"364e435970843c1dd887d44660ad9a1bac6e4e17aa830373fb1fffff70afe251"}
Nov 02 02:03:16.932540 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:16.932510    5639 generic.go:356] "Generic (PLEG): container finished" podID="a1964d9f-bf5f-444f-a781-3284985a726a" containerID="38800407ce79e559141cff532f0255c4521407dd2de4f5d8f80da53652a51df0" exitCode=0
Nov 02 02:03:16.932715 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:16.932590    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657df10-g8cdq" event={"ID":"a1964d9f-bf5f-444f-a781-3284985a726a","Type":"ContainerDied","Data":"38800407ce79e559141cff532f0255c4521407dd2de4f5d8f80da53652a51df0"}
Nov 02 02:03:16.932715 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:16.932624    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657df10-g8cdq" event={"ID":"a1964d9f-bf5f-444f-a781-3284985a726a","Type":"ContainerStarted","Data":"5f12f11e7c65ac49138e54f4f0004323c7ebcf9baecb2bea20baa820134dfc1e"}
Nov 02 02:03:17.052299 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:17.048795    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-537ee113-kwdbr" podStartSLOduration=2.048778318 podStartE2EDuration="2.048778318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:03:17.045132499 +0000 UTC m=+2261.948814613" watchObservedRunningTime="2026-11-02 02:03:17.048778318 +0000 UTC m=+2261.952460432"
Nov 02 02:03:17.942110 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:17.942075    5639 generic.go:356] "Generic (PLEG): container finished" podID="875bffb8-7c4b-44dc-97fe-3be42c9ffe93" containerID="d749c0c7b223a006350800493d57af6b3aa98dc8a0aa525db35b41b81297a0da" exitCode=0
Nov 02 02:03:17.942576 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:17.942154    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-537ee113-kwdbr" event={"ID":"875bffb8-7c4b-44dc-97fe-3be42c9ffe93","Type":"ContainerDied","Data":"d749c0c7b223a006350800493d57af6b3aa98dc8a0aa525db35b41b81297a0da"}
Nov 02 02:03:18.100823 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:18.100799    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657df10-g8cdq"
Nov 02 02:03:18.150317 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:18.150261    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1964d9f-bf5f-444f-a781-3284985a726a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1964d9f-bf5f-444f-a781-3284985a726a-maas-api-openshift-ca\") pod \"a1964d9f-bf5f-444f-a781-3284985a726a\" (UID: \"a1964d9f-bf5f-444f-a781-3284985a726a\") "
Nov 02 02:03:18.150537 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:18.150438    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1964d9f-bf5f-444f-a781-3284985a726a-kube-api-access-wdwf5\" (UniqueName: \"kubernetes.io/projected/a1964d9f-bf5f-444f-a781-3284985a726a-kube-api-access-wdwf5\") pod \"a1964d9f-bf5f-444f-a781-3284985a726a\" (UID: \"a1964d9f-bf5f-444f-a781-3284985a726a\") "
Nov 02 02:03:18.150601 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:18.150568    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1964d9f-bf5f-444f-a781-3284985a726a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1964d9f-bf5f-444f-a781-3284985a726a-maas-api-xks-ca\") pod \"a1964d9f-bf5f-444f-a781-3284985a726a\" (UID: \"a1964d9f-bf5f-444f-a781-3284985a726a\") "
Nov 02 02:03:18.150658 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:18.150627    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1964d9f-bf5f-444f-a781-3284985a726a-maas-api-openshift-ca" pod "a1964d9f-bf5f-444f-a781-3284985a726a" (UID: "a1964d9f-bf5f-444f-a781-3284985a726a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:03:18.151084 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:18.151063    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1964d9f-bf5f-444f-a781-3284985a726a-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:18.153383 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:18.153057    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1964d9f-bf5f-444f-a781-3284985a726a-kube-api-access-wdwf5" pod "a1964d9f-bf5f-444f-a781-3284985a726a" (UID: "a1964d9f-bf5f-444f-a781-3284985a726a"). InnerVolumeSpecName "kube-api-access-wdwf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:18.153383 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:18.153253    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1964d9f-bf5f-444f-a781-3284985a726a-maas-api-xks-ca" pod "a1964d9f-bf5f-444f-a781-3284985a726a" (UID: "a1964d9f-bf5f-444f-a781-3284985a726a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:03:18.251990 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:18.251901    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdwf5\" (UniqueName: \"kubernetes.io/projected/a1964d9f-bf5f-444f-a781-3284985a726a-kube-api-access-wdwf5\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:18.251990 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:18.251936    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1964d9f-bf5f-444f-a781-3284985a726a-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:18.869884 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:18.869856    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4"
Nov 02 02:03:18.870077 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:18.869934    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf"
Nov 02 02:03:18.953654 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:18.953627    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657df10-g8cdq"
Nov 02 02:03:18.954023 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:18.953657    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657df10-g8cdq" event={"ID":"a1964d9f-bf5f-444f-a781-3284985a726a","Type":"ContainerDied","Data":"5f12f11e7c65ac49138e54f4f0004323c7ebcf9baecb2bea20baa820134dfc1e"}
Nov 02 02:03:18.954023 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:18.953734    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f12f11e7c65ac49138e54f4f0004323c7ebcf9baecb2bea20baa820134dfc1e"
Nov 02 02:03:19.133113 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:19.133089    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-537ee113-kwdbr"
Nov 02 02:03:19.263995 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:19.263963    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/875bffb8-7c4b-44dc-97fe-3be42c9ffe93-kube-api-access-pfpls\" (UniqueName: \"kubernetes.io/projected/875bffb8-7c4b-44dc-97fe-3be42c9ffe93-kube-api-access-pfpls\") pod \"875bffb8-7c4b-44dc-97fe-3be42c9ffe93\" (UID: \"875bffb8-7c4b-44dc-97fe-3be42c9ffe93\") "
Nov 02 02:03:19.264173 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:19.264091    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/875bffb8-7c4b-44dc-97fe-3be42c9ffe93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/875bffb8-7c4b-44dc-97fe-3be42c9ffe93-maas-api-openshift-ca\") pod \"875bffb8-7c4b-44dc-97fe-3be42c9ffe93\" (UID: \"875bffb8-7c4b-44dc-97fe-3be42c9ffe93\") "
Nov 02 02:03:19.264173 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:19.264114    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/875bffb8-7c4b-44dc-97fe-3be42c9ffe93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/875bffb8-7c4b-44dc-97fe-3be42c9ffe93-maas-api-xks-ca\") pod \"875bffb8-7c4b-44dc-97fe-3be42c9ffe93\" (UID: \"875bffb8-7c4b-44dc-97fe-3be42c9ffe93\") "
Nov 02 02:03:19.264515 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:19.264489    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/875bffb8-7c4b-44dc-97fe-3be42c9ffe93-maas-api-openshift-ca" pod "875bffb8-7c4b-44dc-97fe-3be42c9ffe93" (UID: "875bffb8-7c4b-44dc-97fe-3be42c9ffe93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:03:19.266178 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:19.266158    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/875bffb8-7c4b-44dc-97fe-3be42c9ffe93-kube-api-access-pfpls" pod "875bffb8-7c4b-44dc-97fe-3be42c9ffe93" (UID: "875bffb8-7c4b-44dc-97fe-3be42c9ffe93"). InnerVolumeSpecName "kube-api-access-pfpls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:19.266250 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:19.266229    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/875bffb8-7c4b-44dc-97fe-3be42c9ffe93-maas-api-xks-ca" pod "875bffb8-7c4b-44dc-97fe-3be42c9ffe93" (UID: "875bffb8-7c4b-44dc-97fe-3be42c9ffe93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:03:19.365197 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:19.365163    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfpls\" (UniqueName: \"kubernetes.io/projected/875bffb8-7c4b-44dc-97fe-3be42c9ffe93-kube-api-access-pfpls\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:19.365197 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:19.365190    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/875bffb8-7c4b-44dc-97fe-3be42c9ffe93-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:19.365197 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:19.365200    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/875bffb8-7c4b-44dc-97fe-3be42c9ffe93-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:19.963397 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:19.963366    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-537ee113-kwdbr"
Nov 02 02:03:19.963914 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:19.963370    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-537ee113-kwdbr" event={"ID":"875bffb8-7c4b-44dc-97fe-3be42c9ffe93","Type":"ContainerDied","Data":"364e435970843c1dd887d44660ad9a1bac6e4e17aa830373fb1fffff70afe251"}
Nov 02 02:03:19.963914 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:19.963497    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="364e435970843c1dd887d44660ad9a1bac6e4e17aa830373fb1fffff70afe251"
Nov 02 02:03:22.337545 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:22.337509    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:03:25.101907 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.101877    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495306e4-5h4xx"]
Nov 02 02:03:25.102490 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.102476    5639 cpu_manager.go:414] "containerMap: removing container" podUID="875bffb8-7c4b-44dc-97fe-3be42c9ffe93" containerName="revoke-keys"
Nov 02 02:03:25.102546 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.102491    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="875bffb8-7c4b-44dc-97fe-3be42c9ffe93" containerName="revoke-keys"
Nov 02 02:03:25.102546 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.102513    5639 cpu_manager.go:414] "containerMap: removing container" podUID="a1964d9f-bf5f-444f-a781-3284985a726a" containerName="revoke-keys"
Nov 02 02:03:25.102546 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.102518    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1964d9f-bf5f-444f-a781-3284985a726a" containerName="revoke-keys"
Nov 02 02:03:25.102664 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.102634    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="875bffb8-7c4b-44dc-97fe-3be42c9ffe93" containerName="revoke-keys"
Nov 02 02:03:25.102664 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.102645    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1964d9f-bf5f-444f-a781-3284985a726a" containerName="revoke-keys"
Nov 02 02:03:25.106267 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.106247    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495306e4-5h4xx"
Nov 02 02:03:25.162368 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.162333    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0933482-xm6kf"]
Nov 02 02:03:25.167328 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.167306    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0933482-xm6kf"
Nov 02 02:03:25.207424 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.207393    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:03:25.207620 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.207515    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:03:25.209857 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.209819    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:03:25.210044 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.210024    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:03:25.210975 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.210950    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495306e4-5h4xx"]
Nov 02 02:03:25.222890 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.222862    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0933482-xm6kf"]
Nov 02 02:03:25.254626 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.254589    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32de37c2-a071-4294-97e1-0e17a83db12c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-495306e4-5h4xx\" (UID: \"32de37c2-a071-4294-97e1-0e17a83db12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495306e4-5h4xx"
Nov 02 02:03:25.254812 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.254734    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0933482-xm6kf\" (UID: \"a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0933482-xm6kf"
Nov 02 02:03:25.254889 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.254871    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0933482-xm6kf\" (UID: \"a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0933482-xm6kf"
Nov 02 02:03:25.254976 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.254959    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk9n2\" (UniqueName: \"kubernetes.io/projected/a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0-kube-api-access-bk9n2\") pod \"maas-api-revoke-sub-e0933482-xm6kf\" (UID: \"a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0933482-xm6kf"
Nov 02 02:03:25.255067 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.255033    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32de37c2-a071-4294-97e1-0e17a83db12c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-495306e4-5h4xx\" (UID: \"32de37c2-a071-4294-97e1-0e17a83db12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495306e4-5h4xx"
Nov 02 02:03:25.255208 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.255100    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98nql\" (UniqueName: \"kubernetes.io/projected/32de37c2-a071-4294-97e1-0e17a83db12c-kube-api-access-98nql\") pod \"maas-api-revoke-sub-495306e4-5h4xx\" (UID: \"32de37c2-a071-4294-97e1-0e17a83db12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495306e4-5h4xx"
Nov 02 02:03:25.356570 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.356489    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0933482-xm6kf\" (UID: \"a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0933482-xm6kf"
Nov 02 02:03:25.356751 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.356720    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk9n2\" (UniqueName: \"kubernetes.io/projected/a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0-kube-api-access-bk9n2\") pod \"maas-api-revoke-sub-e0933482-xm6kf\" (UID: \"a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0933482-xm6kf"
Nov 02 02:03:25.356828 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.356769    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32de37c2-a071-4294-97e1-0e17a83db12c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-495306e4-5h4xx\" (UID: \"32de37c2-a071-4294-97e1-0e17a83db12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495306e4-5h4xx"
Nov 02 02:03:25.356890 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.356836    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98nql\" (UniqueName: \"kubernetes.io/projected/32de37c2-a071-4294-97e1-0e17a83db12c-kube-api-access-98nql\") pod \"maas-api-revoke-sub-495306e4-5h4xx\" (UID: \"32de37c2-a071-4294-97e1-0e17a83db12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495306e4-5h4xx"
Nov 02 02:03:25.356991 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.356977    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32de37c2-a071-4294-97e1-0e17a83db12c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-495306e4-5h4xx\" (UID: \"32de37c2-a071-4294-97e1-0e17a83db12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495306e4-5h4xx"
Nov 02 02:03:25.357063 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.357050    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0933482-xm6kf\" (UID: \"a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0933482-xm6kf"
Nov 02 02:03:25.357858 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.357819    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32de37c2-a071-4294-97e1-0e17a83db12c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-495306e4-5h4xx\" (UID: \"32de37c2-a071-4294-97e1-0e17a83db12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495306e4-5h4xx"
Nov 02 02:03:25.357858 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.357846    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0933482-xm6kf\" (UID: \"a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0933482-xm6kf"
Nov 02 02:03:25.358987 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.358965    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0933482-xm6kf\" (UID: \"a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0933482-xm6kf"
Nov 02 02:03:25.359170 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.359152    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32de37c2-a071-4294-97e1-0e17a83db12c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-495306e4-5h4xx\" (UID: \"32de37c2-a071-4294-97e1-0e17a83db12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495306e4-5h4xx"
Nov 02 02:03:25.417705 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.417673    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk9n2\" (UniqueName: \"kubernetes.io/projected/a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0-kube-api-access-bk9n2\") pod \"maas-api-revoke-sub-e0933482-xm6kf\" (UID: \"a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0933482-xm6kf"
Nov 02 02:03:25.417705 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.417697    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98nql\" (UniqueName: \"kubernetes.io/projected/32de37c2-a071-4294-97e1-0e17a83db12c-kube-api-access-98nql\") pod \"maas-api-revoke-sub-495306e4-5h4xx\" (UID: \"32de37c2-a071-4294-97e1-0e17a83db12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495306e4-5h4xx"
Nov 02 02:03:25.448309 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.448022    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495306e4-5h4xx"
Nov 02 02:03:25.482046 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.482016    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0933482-xm6kf"
Nov 02 02:03:25.854742 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.854713    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495306e4-5h4xx"]
Nov 02 02:03:25.888395 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:03:25.888364    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32de37c2_a071_4294_97e1_0e17a83db12c.slice/crio-164c30cc35cda12ee7bb9bdd66b0f4c808c0764f81d4420f8a80ee092b80caca WatchSource:0}: Error finding container 164c30cc35cda12ee7bb9bdd66b0f4c808c0764f81d4420f8a80ee092b80caca: Status 404 returned error can't find the container with id 164c30cc35cda12ee7bb9bdd66b0f4c808c0764f81d4420f8a80ee092b80caca
Nov 02 02:03:25.900746 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:25.900720    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0933482-xm6kf"]
Nov 02 02:03:25.902209 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:03:25.902187    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c1ca19_5973_405c_9a94_8a1e9d0d8cc0.slice/crio-3ee176d16e24bcbcd327009183fa7d7da56a1ff06fe301e5f1c23cbd762c2d39 WatchSource:0}: Error finding container 3ee176d16e24bcbcd327009183fa7d7da56a1ff06fe301e5f1c23cbd762c2d39: Status 404 returned error can't find the container with id 3ee176d16e24bcbcd327009183fa7d7da56a1ff06fe301e5f1c23cbd762c2d39
Nov 02 02:03:26.089596 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:26.089559    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495306e4-5h4xx" event={"ID":"32de37c2-a071-4294-97e1-0e17a83db12c","Type":"ContainerStarted","Data":"164c30cc35cda12ee7bb9bdd66b0f4c808c0764f81d4420f8a80ee092b80caca"}
Nov 02 02:03:26.090829 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:26.090803    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0933482-xm6kf" event={"ID":"a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0","Type":"ContainerStarted","Data":"3ee176d16e24bcbcd327009183fa7d7da56a1ff06fe301e5f1c23cbd762c2d39"}
Nov 02 02:03:27.098939 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:27.098909    5639 generic.go:356] "Generic (PLEG): container finished" podID="32de37c2-a071-4294-97e1-0e17a83db12c" containerID="faca89117f02a1ab103ac1e1a4e12e76341420fa731322997cf79dd94de9ca1e" exitCode=0
Nov 02 02:03:27.099459 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:27.098998    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495306e4-5h4xx" event={"ID":"32de37c2-a071-4294-97e1-0e17a83db12c","Type":"ContainerDied","Data":"faca89117f02a1ab103ac1e1a4e12e76341420fa731322997cf79dd94de9ca1e"}
Nov 02 02:03:27.103214 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:27.100765    5639 generic.go:356] "Generic (PLEG): container finished" podID="a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0" containerID="7c83f63d3e104b328d502afc2764281cc369479646c0fa7465f436970ec1f6ea" exitCode=0
Nov 02 02:03:27.103214 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:27.100790    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0933482-xm6kf" event={"ID":"a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0","Type":"ContainerDied","Data":"7c83f63d3e104b328d502afc2764281cc369479646c0fa7465f436970ec1f6ea"}
Nov 02 02:03:27.352683 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:27.352598    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-269235-7ns97"]
Nov 02 02:03:27.355272 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:27.355252    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-269235-7ns97"]
Nov 02 02:03:27.420455 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:03:27.420417    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:03:28.287748 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:28.287721    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495306e4-5h4xx"
Nov 02 02:03:28.291082 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:28.291064    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0933482-xm6kf"
Nov 02 02:03:28.406930 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:28.406844    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0-kube-api-access-bk9n2\" (UniqueName: \"kubernetes.io/projected/a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0-kube-api-access-bk9n2\") pod \"a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0\" (UID: \"a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0\") "
Nov 02 02:03:28.406930 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:28.406898    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32de37c2-a071-4294-97e1-0e17a83db12c-kube-api-access-98nql\" (UniqueName: \"kubernetes.io/projected/32de37c2-a071-4294-97e1-0e17a83db12c-kube-api-access-98nql\") pod \"32de37c2-a071-4294-97e1-0e17a83db12c\" (UID: \"32de37c2-a071-4294-97e1-0e17a83db12c\") "
Nov 02 02:03:28.407185 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:28.406933    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0-maas-api-xks-ca\") pod \"a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0\" (UID: \"a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0\") "
Nov 02 02:03:28.407185 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:28.407098    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0-maas-api-openshift-ca\") pod \"a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0\" (UID: \"a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0\") "
Nov 02 02:03:28.407185 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:28.407133    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32de37c2-a071-4294-97e1-0e17a83db12c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32de37c2-a071-4294-97e1-0e17a83db12c-maas-api-xks-ca\") pod \"32de37c2-a071-4294-97e1-0e17a83db12c\" (UID: \"32de37c2-a071-4294-97e1-0e17a83db12c\") "
Nov 02 02:03:28.407394 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:28.407373    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32de37c2-a071-4294-97e1-0e17a83db12c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32de37c2-a071-4294-97e1-0e17a83db12c-maas-api-openshift-ca\") pod \"32de37c2-a071-4294-97e1-0e17a83db12c\" (UID: \"32de37c2-a071-4294-97e1-0e17a83db12c\") "
Nov 02 02:03:28.407539 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:28.407507    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0-maas-api-openshift-ca" pod "a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0" (UID: "a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:03:28.407868 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:28.407846    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32de37c2-a071-4294-97e1-0e17a83db12c-maas-api-openshift-ca" pod "32de37c2-a071-4294-97e1-0e17a83db12c" (UID: "32de37c2-a071-4294-97e1-0e17a83db12c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:03:28.408094 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:28.408074    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:28.408176 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:28.408099    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32de37c2-a071-4294-97e1-0e17a83db12c-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:28.409885 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:28.409855    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0-maas-api-xks-ca" pod "a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0" (UID: "a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:03:28.409885 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:28.409874    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0-kube-api-access-bk9n2" pod "a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0" (UID: "a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0"). InnerVolumeSpecName "kube-api-access-bk9n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:28.409885 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:28.409860    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32de37c2-a071-4294-97e1-0e17a83db12c-kube-api-access-98nql" pod "32de37c2-a071-4294-97e1-0e17a83db12c" (UID: "32de37c2-a071-4294-97e1-0e17a83db12c"). InnerVolumeSpecName "kube-api-access-98nql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:28.410059 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:28.409939    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32de37c2-a071-4294-97e1-0e17a83db12c-maas-api-xks-ca" pod "32de37c2-a071-4294-97e1-0e17a83db12c" (UID: "32de37c2-a071-4294-97e1-0e17a83db12c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:03:28.509022 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:28.508990    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk9n2\" (UniqueName: \"kubernetes.io/projected/a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0-kube-api-access-bk9n2\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:28.509022 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:28.509018    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98nql\" (UniqueName: \"kubernetes.io/projected/32de37c2-a071-4294-97e1-0e17a83db12c-kube-api-access-98nql\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:28.509216 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:28.509030    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:28.509216 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:28.509039    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32de37c2-a071-4294-97e1-0e17a83db12c-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:29.116549 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:29.116523    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495306e4-5h4xx"
Nov 02 02:03:29.116743 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:29.116521    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495306e4-5h4xx" event={"ID":"32de37c2-a071-4294-97e1-0e17a83db12c","Type":"ContainerDied","Data":"164c30cc35cda12ee7bb9bdd66b0f4c808c0764f81d4420f8a80ee092b80caca"}
Nov 02 02:03:29.116743 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:29.116664    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="164c30cc35cda12ee7bb9bdd66b0f4c808c0764f81d4420f8a80ee092b80caca"
Nov 02 02:03:29.118018 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:29.117994    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0933482-xm6kf"
Nov 02 02:03:29.118160 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:29.118043    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0933482-xm6kf" event={"ID":"a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0","Type":"ContainerDied","Data":"3ee176d16e24bcbcd327009183fa7d7da56a1ff06fe301e5f1c23cbd762c2d39"}
Nov 02 02:03:29.118160 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:29.118087    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ee176d16e24bcbcd327009183fa7d7da56a1ff06fe301e5f1c23cbd762c2d39"
Nov 02 02:03:29.220864 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:29.220832    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bd43750-3a7f-45cd-b4dc-57dec370a295" path="/var/lib/kubelet/pods/3bd43750-3a7f-45cd-b4dc-57dec370a295/volumes"
Nov 02 02:03:33.615426 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:33.615382    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm"]
Nov 02 02:03:33.650707 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:33.650668    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69c15d-q9zlm"]
Nov 02 02:03:34.924530 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:34.924491    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz"]
Nov 02 02:03:34.927181 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:34.927148    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5a72-c8xzz"]
Nov 02 02:03:35.219042 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:35.218954    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d3ccdcb-0da5-408d-a15b-a8be4770b4be" path="/var/lib/kubelet/pods/0d3ccdcb-0da5-408d-a15b-a8be4770b4be/volumes"
Nov 02 02:03:35.219495 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:35.219473    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62180ece-d5f2-48a4-8b9f-7cfe4e814121" path="/var/lib/kubelet/pods/62180ece-d5f2-48a4-8b9f-7cfe4e814121/volumes"
Nov 02 02:03:37.076841 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.076802    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"]
Nov 02 02:03:37.077573 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.077555    5639 cpu_manager.go:414] "containerMap: removing container" podUID="32de37c2-a071-4294-97e1-0e17a83db12c" containerName="revoke-keys"
Nov 02 02:03:37.077623 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.077575    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32de37c2-a071-4294-97e1-0e17a83db12c" containerName="revoke-keys"
Nov 02 02:03:37.077623 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.077618    5639 cpu_manager.go:414] "containerMap: removing container" podUID="a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0" containerName="revoke-keys"
Nov 02 02:03:37.077683 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.077624    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0" containerName="revoke-keys"
Nov 02 02:03:37.077732 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.077703    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0" containerName="revoke-keys"
Nov 02 02:03:37.077732 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.077713    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="32de37c2-a071-4294-97e1-0e17a83db12c" containerName="revoke-keys"
Nov 02 02:03:37.082478 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.082459    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.142915 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.142880    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"]
Nov 02 02:03:37.186673 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.186601    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-workload-socket\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.186673 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.186656    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27rsm\" (UniqueName: \"kubernetes.io/projected/0c89cdf8-4ee0-444d-a8a3-8371f375c432-kube-api-access-27rsm\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.186969 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.186718    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-credential-socket\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.186969 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.186802    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-workload-certs\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.186969 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.186873    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-podinfo\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.186969 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.186904    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-data\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.187158 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.186995    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-token\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.187158 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.187051    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-envoy\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.187158 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.187119    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istiod-ca-cert\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.222775 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.221098    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d0fdb9-openshift-default-dockercfg-66rvc\""
Nov 02 02:03:37.288028 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.287997    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-workload-socket\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.288335 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.288040    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27rsm\" (UniqueName: \"kubernetes.io/projected/0c89cdf8-4ee0-444d-a8a3-8371f375c432-kube-api-access-27rsm\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.288335 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.288062    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-credential-socket\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.288335 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.288079    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-workload-certs\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.288335 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.288095    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-podinfo\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.288335 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.288121    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-data\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.288335 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.288179    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-token\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.288700 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.288342    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-envoy\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.288700 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.288384    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istiod-ca-cert\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.288700 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.288502    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-credential-socket\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.288700 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.288511    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-workload-certs\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.289158 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.289131    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-workload-socket\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.289158 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.289148    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istiod-ca-cert\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.289319 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.289208    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-data\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.291033 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.291007    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-envoy\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.291121 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.291069    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-podinfo\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.350626 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.350576    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-token\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.350761 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.350667    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27rsm\" (UniqueName: \"kubernetes.io/projected/0c89cdf8-4ee0-444d-a8a3-8371f375c432-kube-api-access-27rsm\") pod \"e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") " pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.398307 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.398254    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:37.918759 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.918728    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"]
Nov 02 02:03:37.922878 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:03:37.922851    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c89cdf8_4ee0_444d_a8a3_8371f375c432.slice/crio-3c10bf8af21e02577118f70675b24bc143cb1e09ee45798db34124bdc1b84981 WatchSource:0}: Error finding container 3c10bf8af21e02577118f70675b24bc143cb1e09ee45798db34124bdc1b84981: Status 404 returned error can't find the container with id 3c10bf8af21e02577118f70675b24bc143cb1e09ee45798db34124bdc1b84981
Nov 02 02:03:37.925535 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.925502    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:03:37.925620 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.925565    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:03:37.925620 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:37.925594    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:03:38.287513 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:38.287475    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5" event={"ID":"0c89cdf8-4ee0-444d-a8a3-8371f375c432","Type":"ContainerStarted","Data":"5f1ba773868853577afca0f3215588dbd1a7d3afe9196e4c74e77b36ddd6b058"}
Nov 02 02:03:38.287940 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:38.287555    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5" event={"ID":"0c89cdf8-4ee0-444d-a8a3-8371f375c432","Type":"ContainerStarted","Data":"3c10bf8af21e02577118f70675b24bc143cb1e09ee45798db34124bdc1b84981"}
Nov 02 02:03:38.399093 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:38.399059    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:38.404824 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:38.404794    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:38.410130 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:38.410086    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5" podStartSLOduration=1.410072355 podStartE2EDuration="1.410072355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:03:38.407084091 +0000 UTC m=+2283.310766205" watchObservedRunningTime="2026-11-02 02:03:38.410072355 +0000 UTC m=+2283.313754467"
Nov 02 02:03:39.294460 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:39.294429    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:39.295500 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:39.295478    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:03:42.413875 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:42.413840    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"]
Nov 02 02:03:42.414454 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:42.414074    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276" podUID="85c9d8c5-bc73-4b32-b543-adc3d9509702" containerName="istio-proxy" containerID="cri-o://95b450db3daa5325868975e56fc1e70ef6b3d102c887199edd945ee6875fe348" gracePeriod=30
Nov 02 02:03:43.801823 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:43.801778    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"]
Nov 02 02:03:43.808156 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:43.808131    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:43.863201 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:43.863157    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-76e530-openshift-default-dockercfg-2h2n8\""
Nov 02 02:03:43.914724 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:43.914688    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"]
Nov 02 02:03:43.920037 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:43.920012    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49"]
Nov 02 02:03:43.925253 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:43.925223    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49"
Nov 02 02:03:43.926915 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:43.926883    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-envoy\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:43.927019 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:43.926979    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-credential-socket\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:43.927075 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:43.927025    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-token\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:43.927075 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:43.927058    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-workload-socket\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:43.927146 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:43.927117    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-data\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:43.927208 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:43.927176    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-podinfo\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:43.927270 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:43.927254    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b64c8f8-821f-4d7b-a684-40433fb5396c-istiod-ca-cert\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:43.927345 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:43.927311    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6mks\" (UniqueName: \"kubernetes.io/projected/7b64c8f8-821f-4d7b-a684-40433fb5396c-kube-api-access-w6mks\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:43.927396 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:43.927352    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-workload-certs\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:43.979266 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:43.979235    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49"]
Nov 02 02:03:43.979588 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:43.979561    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:03:43.979588 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:43.979580    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:03:43.979731 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:43.979571    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:03:43.979731 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:43.979638    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:03:44.040304 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.040080    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbsrq\" (UniqueName: \"kubernetes.io/projected/95f26f50-842b-4bc8-a059-e838e04ef4f8-kube-api-access-nbsrq\") pod \"maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49\" (UID: \"95f26f50-842b-4bc8-a059-e838e04ef4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49"
Nov 02 02:03:44.040304 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.040185    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-envoy\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:44.040304 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.040268    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-credential-socket\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:44.040635 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.040329    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-token\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:44.040635 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.040383    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95f26f50-842b-4bc8-a059-e838e04ef4f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49\" (UID: \"95f26f50-842b-4bc8-a059-e838e04ef4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49"
Nov 02 02:03:44.040635 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.040415    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-workload-socket\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:44.040635 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.040478    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-data\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:44.040635 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.040522    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95f26f50-842b-4bc8-a059-e838e04ef4f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49\" (UID: \"95f26f50-842b-4bc8-a059-e838e04ef4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49"
Nov 02 02:03:44.040635 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.040562    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-podinfo\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:44.040635 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.040625    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b64c8f8-821f-4d7b-a684-40433fb5396c-istiod-ca-cert\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:44.040937 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.040658    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6mks\" (UniqueName: \"kubernetes.io/projected/7b64c8f8-821f-4d7b-a684-40433fb5396c-kube-api-access-w6mks\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:44.040937 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.040698    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-workload-certs\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:44.062353 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.041199    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-workload-certs\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:44.071052 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.067027    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-credential-socket\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:44.071052 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.067672    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-workload-socket\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:44.071052 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.067896    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-data\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:44.071052 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.068529    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b64c8f8-821f-4d7b-a684-40433fb5396c-istiod-ca-cert\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:44.074622 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.074582    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-podinfo\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:44.078345 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.076494    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-envoy\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:44.130121 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.130081    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-token\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:44.131893 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.131864    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6mks\" (UniqueName: \"kubernetes.io/projected/7b64c8f8-821f-4d7b-a684-40433fb5396c-kube-api-access-w6mks\") pod \"e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") " pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:44.141919 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.141884    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95f26f50-842b-4bc8-a059-e838e04ef4f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49\" (UID: \"95f26f50-842b-4bc8-a059-e838e04ef4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49"
Nov 02 02:03:44.142328 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.142297    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbsrq\" (UniqueName: \"kubernetes.io/projected/95f26f50-842b-4bc8-a059-e838e04ef4f8-kube-api-access-nbsrq\") pod \"maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49\" (UID: \"95f26f50-842b-4bc8-a059-e838e04ef4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49"
Nov 02 02:03:44.142746 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.142721    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95f26f50-842b-4bc8-a059-e838e04ef4f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49\" (UID: \"95f26f50-842b-4bc8-a059-e838e04ef4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49"
Nov 02 02:03:44.144974 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.143961    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95f26f50-842b-4bc8-a059-e838e04ef4f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49\" (UID: \"95f26f50-842b-4bc8-a059-e838e04ef4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49"
Nov 02 02:03:44.153820 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.153782    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95f26f50-842b-4bc8-a059-e838e04ef4f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49\" (UID: \"95f26f50-842b-4bc8-a059-e838e04ef4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49"
Nov 02 02:03:44.207206 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.207168    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbsrq\" (UniqueName: \"kubernetes.io/projected/95f26f50-842b-4bc8-a059-e838e04ef4f8-kube-api-access-nbsrq\") pod \"maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49\" (UID: \"95f26f50-842b-4bc8-a059-e838e04ef4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49"
Nov 02 02:03:44.243954 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.243904    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49"
Nov 02 02:03:44.423025 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.422991    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:44.623661 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.623637    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49"]
Nov 02 02:03:44.626011 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:03:44.625982    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f26f50_842b_4bc8_a059_e838e04ef4f8.slice/crio-ee4e2cfd2c07b9518de91cb0f868c81289485a38b53037219dd3610ab0bd0dbb WatchSource:0}: Error finding container ee4e2cfd2c07b9518de91cb0f868c81289485a38b53037219dd3610ab0bd0dbb: Status 404 returned error can't find the container with id ee4e2cfd2c07b9518de91cb0f868c81289485a38b53037219dd3610ab0bd0dbb
Nov 02 02:03:44.808217 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.808189    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"]
Nov 02 02:03:44.809202 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:03:44.809174    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b64c8f8_821f_4d7b_a684_40433fb5396c.slice/crio-9d49e618bb15632edb3890edbad06f7a43e4042436eaf531cf756ddb49cdd5b3 WatchSource:0}: Error finding container 9d49e618bb15632edb3890edbad06f7a43e4042436eaf531cf756ddb49cdd5b3: Status 404 returned error can't find the container with id 9d49e618bb15632edb3890edbad06f7a43e4042436eaf531cf756ddb49cdd5b3
Nov 02 02:03:44.811984 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.811947    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:03:44.812109 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.812028    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:03:44.812109 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:44.812076    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:03:45.345895 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:45.345851    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w" event={"ID":"7b64c8f8-821f-4d7b-a684-40433fb5396c","Type":"ContainerStarted","Data":"ecf06b1e5351a646431263dc195d597b322c9baee00af15d3f8288fa6b82f872"}
Nov 02 02:03:45.346085 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:45.345925    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w" event={"ID":"7b64c8f8-821f-4d7b-a684-40433fb5396c","Type":"ContainerStarted","Data":"9d49e618bb15632edb3890edbad06f7a43e4042436eaf531cf756ddb49cdd5b3"}
Nov 02 02:03:45.347706 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:45.347673    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49" event={"ID":"95f26f50-842b-4bc8-a059-e838e04ef4f8","Type":"ContainerStarted","Data":"ddaa730d22f71270f7d7343088ca61c1ec8663472d1a1d7a6e1660ec7ebb04e7"}
Nov 02 02:03:45.347826 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:45.347725    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49" event={"ID":"95f26f50-842b-4bc8-a059-e838e04ef4f8","Type":"ContainerStarted","Data":"ee4e2cfd2c07b9518de91cb0f868c81289485a38b53037219dd3610ab0bd0dbb"}
Nov 02 02:03:45.423723 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:45.423693    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:45.429565 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:45.429541    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:45.466197 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:45.466151    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w" podStartSLOduration=2.466137215 podStartE2EDuration="2.466137215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:03:45.464814155 +0000 UTC m=+2290.368496268" watchObservedRunningTime="2026-11-02 02:03:45.466137215 +0000 UTC m=+2290.369819328"
Nov 02 02:03:45.730611 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:45.730500    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49" podStartSLOduration=2.730486377 podStartE2EDuration="2.730486377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:03:45.727242607 +0000 UTC m=+2290.630924721" watchObservedRunningTime="2026-11-02 02:03:45.730486377 +0000 UTC m=+2290.634168503"
Nov 02 02:03:46.358861 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:46.358823    5639 generic.go:356] "Generic (PLEG): container finished" podID="95f26f50-842b-4bc8-a059-e838e04ef4f8" containerID="ddaa730d22f71270f7d7343088ca61c1ec8663472d1a1d7a6e1660ec7ebb04e7" exitCode=0
Nov 02 02:03:46.359345 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:46.358909    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49" event={"ID":"95f26f50-842b-4bc8-a059-e838e04ef4f8","Type":"ContainerDied","Data":"ddaa730d22f71270f7d7343088ca61c1ec8663472d1a1d7a6e1660ec7ebb04e7"}
Nov 02 02:03:46.359464 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:46.359449    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:46.360540 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:46.360519    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:03:47.502815 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.502777    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"]
Nov 02 02:03:47.551056 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.551029    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49"
Nov 02 02:03:47.586001 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.585973    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95f26f50-842b-4bc8-a059-e838e04ef4f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95f26f50-842b-4bc8-a059-e838e04ef4f8-maas-api-openshift-ca\") pod \"95f26f50-842b-4bc8-a059-e838e04ef4f8\" (UID: \"95f26f50-842b-4bc8-a059-e838e04ef4f8\") "
Nov 02 02:03:47.586177 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.586015    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95f26f50-842b-4bc8-a059-e838e04ef4f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95f26f50-842b-4bc8-a059-e838e04ef4f8-maas-api-xks-ca\") pod \"95f26f50-842b-4bc8-a059-e838e04ef4f8\" (UID: \"95f26f50-842b-4bc8-a059-e838e04ef4f8\") "
Nov 02 02:03:47.586177 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.586068    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95f26f50-842b-4bc8-a059-e838e04ef4f8-kube-api-access-nbsrq\" (UniqueName: \"kubernetes.io/projected/95f26f50-842b-4bc8-a059-e838e04ef4f8-kube-api-access-nbsrq\") pod \"95f26f50-842b-4bc8-a059-e838e04ef4f8\" (UID: \"95f26f50-842b-4bc8-a059-e838e04ef4f8\") "
Nov 02 02:03:47.586479 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.586440    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95f26f50-842b-4bc8-a059-e838e04ef4f8-maas-api-openshift-ca" pod "95f26f50-842b-4bc8-a059-e838e04ef4f8" (UID: "95f26f50-842b-4bc8-a059-e838e04ef4f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:03:47.586635 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.586617    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95f26f50-842b-4bc8-a059-e838e04ef4f8-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:47.589128 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.589104    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95f26f50-842b-4bc8-a059-e838e04ef4f8-maas-api-xks-ca" pod "95f26f50-842b-4bc8-a059-e838e04ef4f8" (UID: "95f26f50-842b-4bc8-a059-e838e04ef4f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:03:47.589227 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.589132    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f26f50-842b-4bc8-a059-e838e04ef4f8-kube-api-access-nbsrq" pod "95f26f50-842b-4bc8-a059-e838e04ef4f8" (UID: "95f26f50-842b-4bc8-a059-e838e04ef4f8"). InnerVolumeSpecName "kube-api-access-nbsrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:47.675813 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.675787    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 02:03:47.688324 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.688295    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbsrq\" (UniqueName: \"kubernetes.io/projected/95f26f50-842b-4bc8-a059-e838e04ef4f8-kube-api-access-nbsrq\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:47.688324 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.688322    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95f26f50-842b-4bc8-a059-e838e04ef4f8-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:47.788813 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.788779    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-workload-socket\") pod \"85c9d8c5-bc73-4b32-b543-adc3d9509702\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") "
Nov 02 02:03:47.788995 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.788879    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-workload-certs\") pod \"85c9d8c5-bc73-4b32-b543-adc3d9509702\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") "
Nov 02 02:03:47.788995 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.788905    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-data\") pod \"85c9d8c5-bc73-4b32-b543-adc3d9509702\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") "
Nov 02 02:03:47.788995 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.788971    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-credential-socket\") pod \"85c9d8c5-bc73-4b32-b543-adc3d9509702\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") "
Nov 02 02:03:47.789165 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.789001    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85c9d8c5-bc73-4b32-b543-adc3d9509702-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85c9d8c5-bc73-4b32-b543-adc3d9509702-istiod-ca-cert\") pod \"85c9d8c5-bc73-4b32-b543-adc3d9509702\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") "
Nov 02 02:03:47.789165 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.789046    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-podinfo\") pod \"85c9d8c5-bc73-4b32-b543-adc3d9509702\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") "
Nov 02 02:03:47.789165 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.789077    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-workload-socket" pod "85c9d8c5-bc73-4b32-b543-adc3d9509702" (UID: "85c9d8c5-bc73-4b32-b543-adc3d9509702"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:47.789362 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.789134    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-envoy\") pod \"85c9d8c5-bc73-4b32-b543-adc3d9509702\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") "
Nov 02 02:03:47.789362 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.789217    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-workload-certs" pod "85c9d8c5-bc73-4b32-b543-adc3d9509702" (UID: "85c9d8c5-bc73-4b32-b543-adc3d9509702"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:47.789362 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.789246    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-token\" (UniqueName: \"kubernetes.io/projected/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-token\") pod \"85c9d8c5-bc73-4b32-b543-adc3d9509702\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") "
Nov 02 02:03:47.789362 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.789254    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-data" pod "85c9d8c5-bc73-4b32-b543-adc3d9509702" (UID: "85c9d8c5-bc73-4b32-b543-adc3d9509702"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:47.789362 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.789355    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85c9d8c5-bc73-4b32-b543-adc3d9509702-kube-api-access-vt6t9\" (UniqueName: \"kubernetes.io/projected/85c9d8c5-bc73-4b32-b543-adc3d9509702-kube-api-access-vt6t9\") pod \"85c9d8c5-bc73-4b32-b543-adc3d9509702\" (UID: \"85c9d8c5-bc73-4b32-b543-adc3d9509702\") "
Nov 02 02:03:47.789624 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.789387    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-credential-socket" pod "85c9d8c5-bc73-4b32-b543-adc3d9509702" (UID: "85c9d8c5-bc73-4b32-b543-adc3d9509702"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:47.789624 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.789450    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85c9d8c5-bc73-4b32-b543-adc3d9509702-istiod-ca-cert" pod "85c9d8c5-bc73-4b32-b543-adc3d9509702" (UID: "85c9d8c5-bc73-4b32-b543-adc3d9509702"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:03:47.790141 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.790121    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:47.790228 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.790147    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:47.790228 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.790161    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:47.790228 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.790175    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:47.790228 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.790189    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85c9d8c5-bc73-4b32-b543-adc3d9509702-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:47.791616 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.791590    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85c9d8c5-bc73-4b32-b543-adc3d9509702-kube-api-access-vt6t9" pod "85c9d8c5-bc73-4b32-b543-adc3d9509702" (UID: "85c9d8c5-bc73-4b32-b543-adc3d9509702"). InnerVolumeSpecName "kube-api-access-vt6t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:47.791723 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.791620    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-token" pod "85c9d8c5-bc73-4b32-b543-adc3d9509702" (UID: "85c9d8c5-bc73-4b32-b543-adc3d9509702"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:47.791960 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.791939    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-podinfo" pod "85c9d8c5-bc73-4b32-b543-adc3d9509702" (UID: "85c9d8c5-bc73-4b32-b543-adc3d9509702"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:03:47.791960 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.791947    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-envoy" pod "85c9d8c5-bc73-4b32-b543-adc3d9509702" (UID: "85c9d8c5-bc73-4b32-b543-adc3d9509702"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:47.891495 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.891458    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:47.891495 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.891483    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:47.891495 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.891493    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85c9d8c5-bc73-4b32-b543-adc3d9509702-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:47.891495 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:47.891502    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt6t9\" (UniqueName: \"kubernetes.io/projected/85c9d8c5-bc73-4b32-b543-adc3d9509702-kube-api-access-vt6t9\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:48.387655 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:48.387614    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49" event={"ID":"95f26f50-842b-4bc8-a059-e838e04ef4f8","Type":"ContainerDied","Data":"ee4e2cfd2c07b9518de91cb0f868c81289485a38b53037219dd3610ab0bd0dbb"}
Nov 02 02:03:48.387924 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:48.387629    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49"
Nov 02 02:03:48.388058 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:48.388039    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee4e2cfd2c07b9518de91cb0f868c81289485a38b53037219dd3610ab0bd0dbb"
Nov 02 02:03:48.392062 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:48.391939    5639 generic.go:356] "Generic (PLEG): container finished" podID="85c9d8c5-bc73-4b32-b543-adc3d9509702" containerID="95b450db3daa5325868975e56fc1e70ef6b3d102c887199edd945ee6875fe348" exitCode=0
Nov 02 02:03:48.392463 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:48.392257    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276" event={"ID":"85c9d8c5-bc73-4b32-b543-adc3d9509702","Type":"ContainerDied","Data":"95b450db3daa5325868975e56fc1e70ef6b3d102c887199edd945ee6875fe348"}
Nov 02 02:03:48.392463 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:48.392369    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276" event={"ID":"85c9d8c5-bc73-4b32-b543-adc3d9509702","Type":"ContainerDied","Data":"12fee098277d3b847ac2d7e96636d211429fb9209bceddf194a6ee10a6b80a38"}
Nov 02 02:03:48.392463 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:48.392417    5639 scope.go:122] "RemoveContainer" containerID="95b450db3daa5325868975e56fc1e70ef6b3d102c887199edd945ee6875fe348"
Nov 02 02:03:48.392873 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:48.392647    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"
Nov 02 02:03:48.404489 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:48.404471    5639 scope.go:122] "RemoveContainer" containerID="7307ad5782c6bf3c83a8d5bdf5d74dceb401a2a1c965c4c0615949c242c1e3f6"
Nov 02 02:03:48.414035 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:48.414018    5639 scope.go:122] "RemoveContainer" containerID="95b450db3daa5325868975e56fc1e70ef6b3d102c887199edd945ee6875fe348"
Nov 02 02:03:48.414339 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:03:48.414319    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b450db3daa5325868975e56fc1e70ef6b3d102c887199edd945ee6875fe348\": container with ID starting with 95b450db3daa5325868975e56fc1e70ef6b3d102c887199edd945ee6875fe348 not found: ID does not exist" containerID="95b450db3daa5325868975e56fc1e70ef6b3d102c887199edd945ee6875fe348"
Nov 02 02:03:48.414403 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:48.414349    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b450db3daa5325868975e56fc1e70ef6b3d102c887199edd945ee6875fe348"} err="failed to get container status \"95b450db3daa5325868975e56fc1e70ef6b3d102c887199edd945ee6875fe348\": rpc error: code = NotFound desc = could not find container \"95b450db3daa5325868975e56fc1e70ef6b3d102c887199edd945ee6875fe348\": container with ID starting with 95b450db3daa5325868975e56fc1e70ef6b3d102c887199edd945ee6875fe348 not found: ID does not exist"
Nov 02 02:03:48.414403 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:48.414366    5639 scope.go:122] "RemoveContainer" containerID="7307ad5782c6bf3c83a8d5bdf5d74dceb401a2a1c965c4c0615949c242c1e3f6"
Nov 02 02:03:48.414632 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:03:48.414616    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7307ad5782c6bf3c83a8d5bdf5d74dceb401a2a1c965c4c0615949c242c1e3f6\": container with ID starting with 7307ad5782c6bf3c83a8d5bdf5d74dceb401a2a1c965c4c0615949c242c1e3f6 not found: ID does not exist" containerID="7307ad5782c6bf3c83a8d5bdf5d74dceb401a2a1c965c4c0615949c242c1e3f6"
Nov 02 02:03:48.414683 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:48.414638    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7307ad5782c6bf3c83a8d5bdf5d74dceb401a2a1c965c4c0615949c242c1e3f6"} err="failed to get container status \"7307ad5782c6bf3c83a8d5bdf5d74dceb401a2a1c965c4c0615949c242c1e3f6\": rpc error: code = NotFound desc = could not find container \"7307ad5782c6bf3c83a8d5bdf5d74dceb401a2a1c965c4c0615949c242c1e3f6\": container with ID starting with 7307ad5782c6bf3c83a8d5bdf5d74dceb401a2a1c965c4c0615949c242c1e3f6 not found: ID does not exist"
Nov 02 02:03:48.558333 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:48.558301    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"]
Nov 02 02:03:48.607992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:48.607958    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ef5a72-openshift-default-54ff854b7-kq276"]
Nov 02 02:03:49.220226 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:49.220196    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85c9d8c5-bc73-4b32-b543-adc3d9509702" path="/var/lib/kubelet/pods/85c9d8c5-bc73-4b32-b543-adc3d9509702/volumes"
Nov 02 02:03:57.422243 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:03:57.422198    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:03:57.959755 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:03:57.959721    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2"]
Nov 02 02:04:01.135422 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:01.135389    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7051-b231c5"]
Nov 02 02:04:01.135999 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:01.135984    5639 cpu_manager.go:414] "containerMap: removing container" podUID="95f26f50-842b-4bc8-a059-e838e04ef4f8" containerName="revoke-keys"
Nov 02 02:04:01.136054 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:01.136001    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95f26f50-842b-4bc8-a059-e838e04ef4f8" containerName="revoke-keys"
Nov 02 02:04:01.136054 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:01.136015    5639 cpu_manager.go:414] "containerMap: removing container" podUID="85c9d8c5-bc73-4b32-b543-adc3d9509702" containerName="istio-proxy"
Nov 02 02:04:01.136054 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:01.136021    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85c9d8c5-bc73-4b32-b543-adc3d9509702" containerName="istio-proxy"
Nov 02 02:04:01.136054 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:01.136027    5639 cpu_manager.go:414] "containerMap: removing container" podUID="85c9d8c5-bc73-4b32-b543-adc3d9509702" containerName="istio-proxy"
Nov 02 02:04:01.136054 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:01.136031    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85c9d8c5-bc73-4b32-b543-adc3d9509702" containerName="istio-proxy"
Nov 02 02:04:01.136227 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:01.136141    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="95f26f50-842b-4bc8-a059-e838e04ef4f8" containerName="revoke-keys"
Nov 02 02:04:01.136227 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:01.136150    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="85c9d8c5-bc73-4b32-b543-adc3d9509702" containerName="istio-proxy"
Nov 02 02:04:01.136227 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:01.136157    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="85c9d8c5-bc73-4b32-b543-adc3d9509702" containerName="istio-proxy"
Nov 02 02:04:01.139711 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:01.139686    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7051-b231c5"
Nov 02 02:04:01.184377 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:01.184337    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmw6w\" (UniqueName: \"kubernetes.io/projected/916d28a3-1240-4476-ad27-1b234b1f716c-kube-api-access-vmw6w\") pod \"test-curl-7051-b231c5\" (UID: \"916d28a3-1240-4476-ad27-1b234b1f716c\") " pod="opendatahub/test-curl-7051-b231c5"
Nov 02 02:04:01.194996 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:01.194963    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7051-b231c5"]
Nov 02 02:04:01.240392 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:01.240363    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-n9czn\""
Nov 02 02:04:01.285287 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:01.285248    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmw6w\" (UniqueName: \"kubernetes.io/projected/916d28a3-1240-4476-ad27-1b234b1f716c-kube-api-access-vmw6w\") pod \"test-curl-7051-b231c5\" (UID: \"916d28a3-1240-4476-ad27-1b234b1f716c\") " pod="opendatahub/test-curl-7051-b231c5"
Nov 02 02:04:01.347184 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:01.347144    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmw6w\" (UniqueName: \"kubernetes.io/projected/916d28a3-1240-4476-ad27-1b234b1f716c-kube-api-access-vmw6w\") pod \"test-curl-7051-b231c5\" (UID: \"916d28a3-1240-4476-ad27-1b234b1f716c\") " pod="opendatahub/test-curl-7051-b231c5"
Nov 02 02:04:01.457859 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:01.457773    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7051-b231c5"
Nov 02 02:04:01.836964 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:01.836917    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7051-b231c5"]
Nov 02 02:04:01.838048 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:04:01.838022    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod916d28a3_1240_4476_ad27_1b234b1f716c.slice/crio-4ded0aaef9d80c359377e4bf752260cfb657532a238eaac4f2166e182f692842 WatchSource:0}: Error finding container 4ded0aaef9d80c359377e4bf752260cfb657532a238eaac4f2166e182f692842: Status 404 returned error can't find the container with id 4ded0aaef9d80c359377e4bf752260cfb657532a238eaac4f2166e182f692842
Nov 02 02:04:02.119620 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:02.119534    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:04:02.494768 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:02.494668    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7051-b231c5" event={"ID":"916d28a3-1240-4476-ad27-1b234b1f716c","Type":"ContainerStarted","Data":"d3920a70250455d67636883d1a1783d35c85a4984b83219793e6b19990741bf3"}
Nov 02 02:04:02.494768 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:02.494718    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7051-b231c5" event={"ID":"916d28a3-1240-4476-ad27-1b234b1f716c","Type":"ContainerStarted","Data":"4ded0aaef9d80c359377e4bf752260cfb657532a238eaac4f2166e182f692842"}
Nov 02 02:04:02.526996 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:02.526952    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main" containerID="cri-o://21d5cf11d1e6a02323a20b2b7460b26be6b711d9bbad6302edc720e41a83c3ad" gracePeriod=45
Nov 02 02:04:02.623108 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:02.623041    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7051-b231c5" podStartSLOduration=1.623020349 podStartE2EDuration="1.623020349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:04:02.620601698 +0000 UTC m=+2307.524283815" watchObservedRunningTime="2026-11-02 02:04:02.623020349 +0000 UTC m=+2307.526702464"
Nov 02 02:04:02.815191 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:02.815159    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 02:04:02.952004 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:02.951965    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-tmp-dir\") pod \"4939edba-ee67-41c9-aa16-809fb1fc7116\" (UID: \"4939edba-ee67-41c9-aa16-809fb1fc7116\") "
Nov 02 02:04:02.952200 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:02.952030    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-model-cache\") pod \"4939edba-ee67-41c9-aa16-809fb1fc7116\" (UID: \"4939edba-ee67-41c9-aa16-809fb1fc7116\") "
Nov 02 02:04:02.952200 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:02.952113    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-dshm\") pod \"4939edba-ee67-41c9-aa16-809fb1fc7116\" (UID: \"4939edba-ee67-41c9-aa16-809fb1fc7116\") "
Nov 02 02:04:02.952200 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:02.952164    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4939edba-ee67-41c9-aa16-809fb1fc7116-tls-certs\" (UniqueName: \"kubernetes.io/secret/4939edba-ee67-41c9-aa16-809fb1fc7116-tls-certs\") pod \"4939edba-ee67-41c9-aa16-809fb1fc7116\" (UID: \"4939edba-ee67-41c9-aa16-809fb1fc7116\") "
Nov 02 02:04:02.952420 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:02.952304    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4939edba-ee67-41c9-aa16-809fb1fc7116-kube-api-access-fsjfp\" (UniqueName: \"kubernetes.io/projected/4939edba-ee67-41c9-aa16-809fb1fc7116-kube-api-access-fsjfp\") pod \"4939edba-ee67-41c9-aa16-809fb1fc7116\" (UID: \"4939edba-ee67-41c9-aa16-809fb1fc7116\") "
Nov 02 02:04:02.952420 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:02.952370    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-home\" (UniqueName: \"kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-home\") pod \"4939edba-ee67-41c9-aa16-809fb1fc7116\" (UID: \"4939edba-ee67-41c9-aa16-809fb1fc7116\") "
Nov 02 02:04:02.952420 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:02.952350    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-tmp-dir" pod "4939edba-ee67-41c9-aa16-809fb1fc7116" (UID: "4939edba-ee67-41c9-aa16-809fb1fc7116"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:02.952690 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:02.952456    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-model-cache" pod "4939edba-ee67-41c9-aa16-809fb1fc7116" (UID: "4939edba-ee67-41c9-aa16-809fb1fc7116"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:02.953547 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:02.953069    5639 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-tmp-dir\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:02.953547 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:02.953103    5639 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-model-cache\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:02.953728 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:02.953640    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-home" pod "4939edba-ee67-41c9-aa16-809fb1fc7116" (UID: "4939edba-ee67-41c9-aa16-809fb1fc7116"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:02.955612 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:02.955585    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-dshm" pod "4939edba-ee67-41c9-aa16-809fb1fc7116" (UID: "4939edba-ee67-41c9-aa16-809fb1fc7116"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:02.957154 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:02.957124    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4939edba-ee67-41c9-aa16-809fb1fc7116-kube-api-access-fsjfp" pod "4939edba-ee67-41c9-aa16-809fb1fc7116" (UID: "4939edba-ee67-41c9-aa16-809fb1fc7116"). InnerVolumeSpecName "kube-api-access-fsjfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:04:02.957458 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:02.957431    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4939edba-ee67-41c9-aa16-809fb1fc7116-tls-certs" pod "4939edba-ee67-41c9-aa16-809fb1fc7116" (UID: "4939edba-ee67-41c9-aa16-809fb1fc7116"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:04:02.960537 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:02.960506    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2" podUID="2d2ed806-9940-4872-93b2-0cd86f5aeb52" containerName="payload-pre-processing" containerID="cri-o://91d58044a9e6dc09bfb24c95c48c3b1a073ca0ac0b669fe87a092960fcca328f" gracePeriod=25
Nov 02 02:04:03.054876 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:03.054780    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsjfp\" (UniqueName: \"kubernetes.io/projected/4939edba-ee67-41c9-aa16-809fb1fc7116-kube-api-access-fsjfp\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:03.054876 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:03.054813    5639 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-home\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:03.054876 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:03.054827    5639 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4939edba-ee67-41c9-aa16-809fb1fc7116-dshm\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:03.054876 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:03.054835    5639 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4939edba-ee67-41c9-aa16-809fb1fc7116-tls-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:03.503060 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:03.503022    5639 generic.go:356] "Generic (PLEG): container finished" podID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerID="21d5cf11d1e6a02323a20b2b7460b26be6b711d9bbad6302edc720e41a83c3ad" exitCode=0
Nov 02 02:04:03.503575 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:03.503086    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"
Nov 02 02:04:03.503575 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:03.503092    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" event={"ID":"4939edba-ee67-41c9-aa16-809fb1fc7116","Type":"ContainerDied","Data":"21d5cf11d1e6a02323a20b2b7460b26be6b711d9bbad6302edc720e41a83c3ad"}
Nov 02 02:04:03.503575 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:03.503149    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77" event={"ID":"4939edba-ee67-41c9-aa16-809fb1fc7116","Type":"ContainerDied","Data":"eba1e026998c97557e95f0fef7dc1b2c1159ce75898eae615797eeb5a77c1556"}
Nov 02 02:04:03.503575 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:03.503170    5639 scope.go:122] "RemoveContainer" containerID="21d5cf11d1e6a02323a20b2b7460b26be6b711d9bbad6302edc720e41a83c3ad"
Nov 02 02:04:03.513733 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:03.513706    5639 scope.go:122] "RemoveContainer" containerID="21d5cf11d1e6a02323a20b2b7460b26be6b711d9bbad6302edc720e41a83c3ad"
Nov 02 02:04:03.514045 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:04:03.514011    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21d5cf11d1e6a02323a20b2b7460b26be6b711d9bbad6302edc720e41a83c3ad\": container with ID starting with 21d5cf11d1e6a02323a20b2b7460b26be6b711d9bbad6302edc720e41a83c3ad not found: ID does not exist" containerID="21d5cf11d1e6a02323a20b2b7460b26be6b711d9bbad6302edc720e41a83c3ad"
Nov 02 02:04:03.514123 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:03.514053    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d5cf11d1e6a02323a20b2b7460b26be6b711d9bbad6302edc720e41a83c3ad"} err="failed to get container status \"21d5cf11d1e6a02323a20b2b7460b26be6b711d9bbad6302edc720e41a83c3ad\": rpc error: code = NotFound desc = could not find container \"21d5cf11d1e6a02323a20b2b7460b26be6b711d9bbad6302edc720e41a83c3ad\": container with ID starting with 21d5cf11d1e6a02323a20b2b7460b26be6b711d9bbad6302edc720e41a83c3ad not found: ID does not exist"
Nov 02 02:04:03.668796 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:03.668769    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"]
Nov 02 02:04:03.673553 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:03.673531    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-f276c3/distinct-w1-f276c3-kserve-7d8f59cfc9-psw77"]
Nov 02 02:04:04.145735 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:04.145693    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7051-b231c5"]
Nov 02 02:04:04.150673 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:04.150646    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7051-b231c5"]
Nov 02 02:04:04.511439 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:04.511337    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7051-b231c5" podUID="916d28a3-1240-4476-ad27-1b234b1f716c" containerName="test-curl-7051-b231c5" containerID="cri-o://d3920a70250455d67636883d1a1783d35c85a4984b83219793e6b19990741bf3" gracePeriod=2
Nov 02 02:04:04.562476 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:04:04.562443    5639 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7051-b231c5\" is forbidden: User \"system:node:ip-10-0-136-53.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-53.us-west-1.compute.internal' and this object" podUID="916d28a3-1240-4476-ad27-1b234b1f716c" pod="opendatahub/test-curl-7051-b231c5"
Nov 02 02:04:04.698111 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:04.698066    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf"]
Nov 02 02:04:04.698990 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:04.698971    5639 cpu_manager.go:414] "containerMap: removing container" podUID="916d28a3-1240-4476-ad27-1b234b1f716c" containerName="test-curl-7051-b231c5"
Nov 02 02:04:04.699068 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:04.698994    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="916d28a3-1240-4476-ad27-1b234b1f716c" containerName="test-curl-7051-b231c5"
Nov 02 02:04:04.699068 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:04.699024    5639 cpu_manager.go:414] "containerMap: removing container" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main"
Nov 02 02:04:04.699068 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:04.699032    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main"
Nov 02 02:04:04.699209 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:04.699198    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="916d28a3-1240-4476-ad27-1b234b1f716c" containerName="test-curl-7051-b231c5"
Nov 02 02:04:04.699245 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:04.699219    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" containerName="main"
Nov 02 02:04:04.704429 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:04.704411    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf"
Nov 02 02:04:04.757862 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:04.757819    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf"]
Nov 02 02:04:04.758111 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:04.758078    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:04:04.758111 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:04.758095    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:04:04.758354 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:04:04.758097    5639 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7051-b231c5\" is forbidden: User \"system:node:ip-10-0-136-53.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-53.us-west-1.compute.internal' and this object" podUID="916d28a3-1240-4476-ad27-1b234b1f716c" pod="opendatahub/test-curl-7051-b231c5"
Nov 02 02:04:04.758354 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:04.758155    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:04:04.758354 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:04.758226    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:04:04.781854 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:04.781766    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fc9915-885c-4a8e-97a4-298fa66ab8aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf\" (UID: \"f7fc9915-885c-4a8e-97a4-298fa66ab8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf"
Nov 02 02:04:04.781854 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:04.781821    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5qk7\" (UniqueName: \"kubernetes.io/projected/f7fc9915-885c-4a8e-97a4-298fa66ab8aa-kube-api-access-m5qk7\") pod \"maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf\" (UID: \"f7fc9915-885c-4a8e-97a4-298fa66ab8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf"
Nov 02 02:04:04.782034 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:04.781963    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fc9915-885c-4a8e-97a4-298fa66ab8aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf\" (UID: \"f7fc9915-885c-4a8e-97a4-298fa66ab8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf"
Nov 02 02:04:04.883381 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:04.883348    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fc9915-885c-4a8e-97a4-298fa66ab8aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf\" (UID: \"f7fc9915-885c-4a8e-97a4-298fa66ab8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf"
Nov 02 02:04:04.883563 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:04.883509    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fc9915-885c-4a8e-97a4-298fa66ab8aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf\" (UID: \"f7fc9915-885c-4a8e-97a4-298fa66ab8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf"
Nov 02 02:04:04.883563 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:04.883538    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5qk7\" (UniqueName: \"kubernetes.io/projected/f7fc9915-885c-4a8e-97a4-298fa66ab8aa-kube-api-access-m5qk7\") pod \"maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf\" (UID: \"f7fc9915-885c-4a8e-97a4-298fa66ab8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf"
Nov 02 02:04:04.884024 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:04.883996    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fc9915-885c-4a8e-97a4-298fa66ab8aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf\" (UID: \"f7fc9915-885c-4a8e-97a4-298fa66ab8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf"
Nov 02 02:04:04.885973 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:04.885952    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fc9915-885c-4a8e-97a4-298fa66ab8aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf\" (UID: \"f7fc9915-885c-4a8e-97a4-298fa66ab8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf"
Nov 02 02:04:04.953869 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:04.953832    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5qk7\" (UniqueName: \"kubernetes.io/projected/f7fc9915-885c-4a8e-97a4-298fa66ab8aa-kube-api-access-m5qk7\") pod \"maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf\" (UID: \"f7fc9915-885c-4a8e-97a4-298fa66ab8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf"
Nov 02 02:04:05.019611 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:05.019577    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf"
Nov 02 02:04:05.219326 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:05.219246    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4939edba-ee67-41c9-aa16-809fb1fc7116" path="/var/lib/kubelet/pods/4939edba-ee67-41c9-aa16-809fb1fc7116/volumes"
Nov 02 02:04:05.266138 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:04:05.266099    5639 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7051-b231c5\" is forbidden: User \"system:node:ip-10-0-136-53.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-53.us-west-1.compute.internal' and this object" podUID="916d28a3-1240-4476-ad27-1b234b1f716c" pod="opendatahub/test-curl-7051-b231c5"
Nov 02 02:04:05.400970 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:05.400938    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf"]
Nov 02 02:04:05.402485 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:04:05.402459    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7fc9915_885c_4a8e_97a4_298fa66ab8aa.slice/crio-5d307fea7cbd7e28dc6c0d31e4610b56e6d65e6df90dc2eac0119f3870a25729 WatchSource:0}: Error finding container 5d307fea7cbd7e28dc6c0d31e4610b56e6d65e6df90dc2eac0119f3870a25729: Status 404 returned error can't find the container with id 5d307fea7cbd7e28dc6c0d31e4610b56e6d65e6df90dc2eac0119f3870a25729
Nov 02 02:04:05.519617 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:05.519580    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf" event={"ID":"f7fc9915-885c-4a8e-97a4-298fa66ab8aa","Type":"ContainerStarted","Data":"5d307fea7cbd7e28dc6c0d31e4610b56e6d65e6df90dc2eac0119f3870a25729"}
Nov 02 02:04:06.556569 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:06.556532    5639 generic.go:356] "Generic (PLEG): container finished" podID="f7fc9915-885c-4a8e-97a4-298fa66ab8aa" containerID="86dd24898d4b287faf71ca7a899d3e5ab3884db504b06775707405819f4d9a55" exitCode=0
Nov 02 02:04:06.556954 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:06.556585    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf" event={"ID":"f7fc9915-885c-4a8e-97a4-298fa66ab8aa","Type":"ContainerDied","Data":"86dd24898d4b287faf71ca7a899d3e5ab3884db504b06775707405819f4d9a55"}
Nov 02 02:04:06.760511 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:06.760474    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm"]
Nov 02 02:04:06.764921 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:06.764896    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm"
Nov 02 02:04:06.815553 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:06.815526    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7051-b231c5"
Nov 02 02:04:06.819919 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:06.819892    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0c954d\""
Nov 02 02:04:06.820164 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:06.819897    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0c954d-dockercfg-28h6j\""
Nov 02 02:04:06.824620 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:06.824581    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm"]
Nov 02 02:04:06.843927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:06.843884    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e08af262-d376-4399-8a42-99bd89b2897c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm\" (UID: \"e08af262-d376-4399-8a42-99bd89b2897c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm"
Nov 02 02:04:06.844100 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:06.843959    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpnqs\" (UniqueName: \"kubernetes.io/projected/e08af262-d376-4399-8a42-99bd89b2897c-kube-api-access-hpnqs\") pod \"payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm\" (UID: \"e08af262-d376-4399-8a42-99bd89b2897c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm"
Nov 02 02:04:06.948138 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:06.946859    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/916d28a3-1240-4476-ad27-1b234b1f716c-kube-api-access-vmw6w\" (UniqueName: \"kubernetes.io/projected/916d28a3-1240-4476-ad27-1b234b1f716c-kube-api-access-vmw6w\") pod \"916d28a3-1240-4476-ad27-1b234b1f716c\" (UID: \"916d28a3-1240-4476-ad27-1b234b1f716c\") "
Nov 02 02:04:06.948138 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:06.947420    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e08af262-d376-4399-8a42-99bd89b2897c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm\" (UID: \"e08af262-d376-4399-8a42-99bd89b2897c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm"
Nov 02 02:04:06.948138 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:06.947483    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpnqs\" (UniqueName: \"kubernetes.io/projected/e08af262-d376-4399-8a42-99bd89b2897c-kube-api-access-hpnqs\") pod \"payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm\" (UID: \"e08af262-d376-4399-8a42-99bd89b2897c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm"
Nov 02 02:04:06.949863 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:06.949486    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e08af262-d376-4399-8a42-99bd89b2897c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm\" (UID: \"e08af262-d376-4399-8a42-99bd89b2897c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm"
Nov 02 02:04:06.975921 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:06.975662    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916d28a3-1240-4476-ad27-1b234b1f716c-kube-api-access-vmw6w" pod "916d28a3-1240-4476-ad27-1b234b1f716c" (UID: "916d28a3-1240-4476-ad27-1b234b1f716c"). InnerVolumeSpecName "kube-api-access-vmw6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:04:06.998680 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:06.998436    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl"]
Nov 02 02:04:07.003369 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.003313    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl"]
Nov 02 02:04:07.003547 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.003405    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl"
Nov 02 02:04:07.008491 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.008447    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpnqs\" (UniqueName: \"kubernetes.io/projected/e08af262-d376-4399-8a42-99bd89b2897c-kube-api-access-hpnqs\") pod \"payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm\" (UID: \"e08af262-d376-4399-8a42-99bd89b2897c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm"
Nov 02 02:04:07.048947 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.048907    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb1c148-f351-4295-892b-03ae51db8912-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl\" (UID: \"9bb1c148-f351-4295-892b-03ae51db8912\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl"
Nov 02 02:04:07.049106 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.048959    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j59rk\" (UniqueName: \"kubernetes.io/projected/9bb1c148-f351-4295-892b-03ae51db8912-kube-api-access-j59rk\") pod \"payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl\" (UID: \"9bb1c148-f351-4295-892b-03ae51db8912\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl"
Nov 02 02:04:07.049251 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.049231    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmw6w\" (UniqueName: \"kubernetes.io/projected/916d28a3-1240-4476-ad27-1b234b1f716c-kube-api-access-vmw6w\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:07.103422 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.103382    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm"
Nov 02 02:04:07.150726 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.150680    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb1c148-f351-4295-892b-03ae51db8912-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl\" (UID: \"9bb1c148-f351-4295-892b-03ae51db8912\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl"
Nov 02 02:04:07.150905 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.150751    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j59rk\" (UniqueName: \"kubernetes.io/projected/9bb1c148-f351-4295-892b-03ae51db8912-kube-api-access-j59rk\") pod \"payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl\" (UID: \"9bb1c148-f351-4295-892b-03ae51db8912\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl"
Nov 02 02:04:07.152266 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.152234    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb1c148-f351-4295-892b-03ae51db8912-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl\" (UID: \"9bb1c148-f351-4295-892b-03ae51db8912\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl"
Nov 02 02:04:07.212177 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.212144    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j59rk\" (UniqueName: \"kubernetes.io/projected/9bb1c148-f351-4295-892b-03ae51db8912-kube-api-access-j59rk\") pod \"payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl\" (UID: \"9bb1c148-f351-4295-892b-03ae51db8912\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl"
Nov 02 02:04:07.222666 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.222581    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="916d28a3-1240-4476-ad27-1b234b1f716c" path="/var/lib/kubelet/pods/916d28a3-1240-4476-ad27-1b234b1f716c/volumes"
Nov 02 02:04:07.339526 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.339489    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl"
Nov 02 02:04:07.488171 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.488145    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm"]
Nov 02 02:04:07.489673 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:04:07.489641    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08af262_d376_4399_8a42_99bd89b2897c.slice/crio-da2bf7b5aa22beff5d271137096e2e4ba7d70ce104f6a704ea2fcc1713a27732 WatchSource:0}: Error finding container da2bf7b5aa22beff5d271137096e2e4ba7d70ce104f6a704ea2fcc1713a27732: Status 404 returned error can't find the container with id da2bf7b5aa22beff5d271137096e2e4ba7d70ce104f6a704ea2fcc1713a27732
Nov 02 02:04:07.565045 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.565012    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm" event={"ID":"e08af262-d376-4399-8a42-99bd89b2897c","Type":"ContainerStarted","Data":"da2bf7b5aa22beff5d271137096e2e4ba7d70ce104f6a704ea2fcc1713a27732"}
Nov 02 02:04:07.566302 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.566254    5639 generic.go:356] "Generic (PLEG): container finished" podID="916d28a3-1240-4476-ad27-1b234b1f716c" containerID="d3920a70250455d67636883d1a1783d35c85a4984b83219793e6b19990741bf3" exitCode=137
Nov 02 02:04:07.566377 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.566317    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7051-b231c5"
Nov 02 02:04:07.566377 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.566334    5639 scope.go:122] "RemoveContainer" containerID="d3920a70250455d67636883d1a1783d35c85a4984b83219793e6b19990741bf3"
Nov 02 02:04:07.578156 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.578129    5639 scope.go:122] "RemoveContainer" containerID="d3920a70250455d67636883d1a1783d35c85a4984b83219793e6b19990741bf3"
Nov 02 02:04:07.578578 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:04:07.578549    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3920a70250455d67636883d1a1783d35c85a4984b83219793e6b19990741bf3\": container with ID starting with d3920a70250455d67636883d1a1783d35c85a4984b83219793e6b19990741bf3 not found: ID does not exist" containerID="d3920a70250455d67636883d1a1783d35c85a4984b83219793e6b19990741bf3"
Nov 02 02:04:07.578662 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.578590    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3920a70250455d67636883d1a1783d35c85a4984b83219793e6b19990741bf3"} err="failed to get container status \"d3920a70250455d67636883d1a1783d35c85a4984b83219793e6b19990741bf3\": rpc error: code = NotFound desc = could not find container \"d3920a70250455d67636883d1a1783d35c85a4984b83219793e6b19990741bf3\": container with ID starting with d3920a70250455d67636883d1a1783d35c85a4984b83219793e6b19990741bf3 not found: ID does not exist"
Nov 02 02:04:07.710515 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.710489    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf"
Nov 02 02:04:07.715977 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.715954    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl"]
Nov 02 02:04:07.717855 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:04:07.717826    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb1c148_f351_4295_892b_03ae51db8912.slice/crio-a08fde6f731b935242b1e5ac571b70ed73a90c52926aee5107771274d8e4a498 WatchSource:0}: Error finding container a08fde6f731b935242b1e5ac571b70ed73a90c52926aee5107771274d8e4a498: Status 404 returned error can't find the container with id a08fde6f731b935242b1e5ac571b70ed73a90c52926aee5107771274d8e4a498
Nov 02 02:04:07.758409 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.758384    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7fc9915-885c-4a8e-97a4-298fa66ab8aa-kube-api-access-m5qk7\" (UniqueName: \"kubernetes.io/projected/f7fc9915-885c-4a8e-97a4-298fa66ab8aa-kube-api-access-m5qk7\") pod \"f7fc9915-885c-4a8e-97a4-298fa66ab8aa\" (UID: \"f7fc9915-885c-4a8e-97a4-298fa66ab8aa\") "
Nov 02 02:04:07.758600 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.758584    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7fc9915-885c-4a8e-97a4-298fa66ab8aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fc9915-885c-4a8e-97a4-298fa66ab8aa-maas-api-xks-ca\") pod \"f7fc9915-885c-4a8e-97a4-298fa66ab8aa\" (UID: \"f7fc9915-885c-4a8e-97a4-298fa66ab8aa\") "
Nov 02 02:04:07.758649 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.758616    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7fc9915-885c-4a8e-97a4-298fa66ab8aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fc9915-885c-4a8e-97a4-298fa66ab8aa-maas-api-openshift-ca\") pod \"f7fc9915-885c-4a8e-97a4-298fa66ab8aa\" (UID: \"f7fc9915-885c-4a8e-97a4-298fa66ab8aa\") "
Nov 02 02:04:07.759043 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.759014    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7fc9915-885c-4a8e-97a4-298fa66ab8aa-maas-api-openshift-ca" pod "f7fc9915-885c-4a8e-97a4-298fa66ab8aa" (UID: "f7fc9915-885c-4a8e-97a4-298fa66ab8aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:04:07.759160 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.759136    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fc9915-885c-4a8e-97a4-298fa66ab8aa-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:07.761027 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.760998    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7fc9915-885c-4a8e-97a4-298fa66ab8aa-kube-api-access-m5qk7" pod "f7fc9915-885c-4a8e-97a4-298fa66ab8aa" (UID: "f7fc9915-885c-4a8e-97a4-298fa66ab8aa"). InnerVolumeSpecName "kube-api-access-m5qk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:04:07.761171 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.761149    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7fc9915-885c-4a8e-97a4-298fa66ab8aa-maas-api-xks-ca" pod "f7fc9915-885c-4a8e-97a4-298fa66ab8aa" (UID: "f7fc9915-885c-4a8e-97a4-298fa66ab8aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:04:07.860861 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.860799    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fc9915-885c-4a8e-97a4-298fa66ab8aa-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:07.860861 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:07.860861    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5qk7\" (UniqueName: \"kubernetes.io/projected/f7fc9915-885c-4a8e-97a4-298fa66ab8aa-kube-api-access-m5qk7\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:08.576414 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:08.576361    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf"
Nov 02 02:04:08.576849 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:08.576387    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf" event={"ID":"f7fc9915-885c-4a8e-97a4-298fa66ab8aa","Type":"ContainerDied","Data":"5d307fea7cbd7e28dc6c0d31e4610b56e6d65e6df90dc2eac0119f3870a25729"}
Nov 02 02:04:08.576849 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:08.576520    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d307fea7cbd7e28dc6c0d31e4610b56e6d65e6df90dc2eac0119f3870a25729"
Nov 02 02:04:08.578187 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:08.578151    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl" event={"ID":"9bb1c148-f351-4295-892b-03ae51db8912","Type":"ContainerStarted","Data":"a08e953832d201390379266066b625de2908521d1ba35f03fbad46878229dab7"}
Nov 02 02:04:08.578391 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:08.578373    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl"
Nov 02 02:04:08.578483 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:08.578413    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl" event={"ID":"9bb1c148-f351-4295-892b-03ae51db8912","Type":"ContainerStarted","Data":"a08fde6f731b935242b1e5ac571b70ed73a90c52926aee5107771274d8e4a498"}
Nov 02 02:04:08.579826 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:08.579805    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm" event={"ID":"e08af262-d376-4399-8a42-99bd89b2897c","Type":"ContainerStarted","Data":"7697a99f01b6478cbf5d3ca1f03e12ff45e5863f2b479ae303cdff69527f0e35"}
Nov 02 02:04:08.579958 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:08.579940    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm"
Nov 02 02:04:08.695092 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:08.695028    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl" podStartSLOduration=2.695013129 podStartE2EDuration="2.695013129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:04:08.693464939 +0000 UTC m=+2313.597147054" watchObservedRunningTime="2026-11-02 02:04:08.695013129 +0000 UTC m=+2313.598695244"
Nov 02 02:04:08.850048 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:08.849988    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm" podStartSLOduration=2.849969988 podStartE2EDuration="2.849969988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:04:08.821720093 +0000 UTC m=+2313.725402208" watchObservedRunningTime="2026-11-02 02:04:08.849969988 +0000 UTC m=+2313.753652104"
Nov 02 02:04:09.558910 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:09.558876    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2" podUID="2d2ed806-9940-4872-93b2-0cd86f5aeb52" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:04:15.068413 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:15.068375    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4"]
Nov 02 02:04:15.178321 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:15.175519    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf"]
Nov 02 02:04:17.588389 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:17.588349    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9"]
Nov 02 02:04:17.636176 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:17.636141    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b71ba5-djsr9"]
Nov 02 02:04:19.220008 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:19.219973    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0c26c59-2897-4252-b545-9da191ac7735" path="/var/lib/kubelet/pods/a0c26c59-2897-4252-b545-9da191ac7735/volumes"
Nov 02 02:04:19.558751 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:19.558669    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2" podUID="2d2ed806-9940-4872-93b2-0cd86f5aeb52" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:04:19.591849 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:19.591818    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm"
Nov 02 02:04:19.592026 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:19.591906    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl"
Nov 02 02:04:20.069292 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.069246    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4" podUID="91fe93cf-734d-4a19-868d-b4c38a76bc2b" containerName="payload-processing" containerID="cri-o://20ad0b7f82df36994903072d915deaef626076d57ad891ac85882d0b6eecebce" gracePeriod=25
Nov 02 02:04:20.175971 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.175934    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf" podUID="322c165c-ab83-4cc0-91f2-d83533aa6472" containerName="payload-pre-processing" containerID="cri-o://f6ea5b2fbb2ce852bb231bbb9131c2b5b3c248c91a8d9f6b70fdb9da31e99389" gracePeriod=25
Nov 02 02:04:20.353851 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.353819    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4"
Nov 02 02:04:20.446396 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.446356    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf"
Nov 02 02:04:20.455664 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.455643    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91fe93cf-734d-4a19-868d-b4c38a76bc2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91fe93cf-734d-4a19-868d-b4c38a76bc2b-plugins-config-volume\") pod \"91fe93cf-734d-4a19-868d-b4c38a76bc2b\" (UID: \"91fe93cf-734d-4a19-868d-b4c38a76bc2b\") "
Nov 02 02:04:20.455787 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.455721    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91fe93cf-734d-4a19-868d-b4c38a76bc2b-kube-api-access-l2bjc\" (UniqueName: \"kubernetes.io/projected/91fe93cf-734d-4a19-868d-b4c38a76bc2b-kube-api-access-l2bjc\") pod \"91fe93cf-734d-4a19-868d-b4c38a76bc2b\" (UID: \"91fe93cf-734d-4a19-868d-b4c38a76bc2b\") "
Nov 02 02:04:20.456042 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.456020    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91fe93cf-734d-4a19-868d-b4c38a76bc2b-plugins-config-volume" pod "91fe93cf-734d-4a19-868d-b4c38a76bc2b" (UID: "91fe93cf-734d-4a19-868d-b4c38a76bc2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:04:20.457965 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.457945    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91fe93cf-734d-4a19-868d-b4c38a76bc2b-kube-api-access-l2bjc" pod "91fe93cf-734d-4a19-868d-b4c38a76bc2b" (UID: "91fe93cf-734d-4a19-868d-b4c38a76bc2b"). InnerVolumeSpecName "kube-api-access-l2bjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:04:20.557581 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.557544    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/322c165c-ab83-4cc0-91f2-d83533aa6472-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/322c165c-ab83-4cc0-91f2-d83533aa6472-plugins-config-volume\") pod \"322c165c-ab83-4cc0-91f2-d83533aa6472\" (UID: \"322c165c-ab83-4cc0-91f2-d83533aa6472\") "
Nov 02 02:04:20.557794 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.557664    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/322c165c-ab83-4cc0-91f2-d83533aa6472-kube-api-access-gk2sm\" (UniqueName: \"kubernetes.io/projected/322c165c-ab83-4cc0-91f2-d83533aa6472-kube-api-access-gk2sm\") pod \"322c165c-ab83-4cc0-91f2-d83533aa6472\" (UID: \"322c165c-ab83-4cc0-91f2-d83533aa6472\") "
Nov 02 02:04:20.558010 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.557983    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322c165c-ab83-4cc0-91f2-d83533aa6472-plugins-config-volume" pod "322c165c-ab83-4cc0-91f2-d83533aa6472" (UID: "322c165c-ab83-4cc0-91f2-d83533aa6472"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:04:20.558180 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.558164    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91fe93cf-734d-4a19-868d-b4c38a76bc2b-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:20.558249 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.558185    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2bjc\" (UniqueName: \"kubernetes.io/projected/91fe93cf-734d-4a19-868d-b4c38a76bc2b-kube-api-access-l2bjc\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:20.558249 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.558199    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/322c165c-ab83-4cc0-91f2-d83533aa6472-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:20.559966 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.559943    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322c165c-ab83-4cc0-91f2-d83533aa6472-kube-api-access-gk2sm" pod "322c165c-ab83-4cc0-91f2-d83533aa6472" (UID: "322c165c-ab83-4cc0-91f2-d83533aa6472"). InnerVolumeSpecName "kube-api-access-gk2sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:04:20.659380 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.659342    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk2sm\" (UniqueName: \"kubernetes.io/projected/322c165c-ab83-4cc0-91f2-d83533aa6472-kube-api-access-gk2sm\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:20.714107 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.714067    5639 generic.go:356] "Generic (PLEG): container finished" podID="322c165c-ab83-4cc0-91f2-d83533aa6472" containerID="f6ea5b2fbb2ce852bb231bbb9131c2b5b3c248c91a8d9f6b70fdb9da31e99389" exitCode=0
Nov 02 02:04:20.714331 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.714119    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf"
Nov 02 02:04:20.714331 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.714155    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf" event={"ID":"322c165c-ab83-4cc0-91f2-d83533aa6472","Type":"ContainerDied","Data":"f6ea5b2fbb2ce852bb231bbb9131c2b5b3c248c91a8d9f6b70fdb9da31e99389"}
Nov 02 02:04:20.714331 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.714225    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf" event={"ID":"322c165c-ab83-4cc0-91f2-d83533aa6472","Type":"ContainerDied","Data":"321b1de947a23be8195463a174bfe8795d8abcda8a80328b6cf98b68943a666a"}
Nov 02 02:04:20.714331 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.714253    5639 scope.go:122] "RemoveContainer" containerID="f6ea5b2fbb2ce852bb231bbb9131c2b5b3c248c91a8d9f6b70fdb9da31e99389"
Nov 02 02:04:20.715816 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.715784    5639 generic.go:356] "Generic (PLEG): container finished" podID="91fe93cf-734d-4a19-868d-b4c38a76bc2b" containerID="20ad0b7f82df36994903072d915deaef626076d57ad891ac85882d0b6eecebce" exitCode=0
Nov 02 02:04:20.715942 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.715846    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4"
Nov 02 02:04:20.715942 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.715841    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4" event={"ID":"91fe93cf-734d-4a19-868d-b4c38a76bc2b","Type":"ContainerDied","Data":"20ad0b7f82df36994903072d915deaef626076d57ad891ac85882d0b6eecebce"}
Nov 02 02:04:20.716046 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.715966    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4" event={"ID":"91fe93cf-734d-4a19-868d-b4c38a76bc2b","Type":"ContainerDied","Data":"69c93929ee085c681833d9d480106a40fbbd090a5ec4692cdd10877e5e309748"}
Nov 02 02:04:20.726218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.726194    5639 scope.go:122] "RemoveContainer" containerID="f6ea5b2fbb2ce852bb231bbb9131c2b5b3c248c91a8d9f6b70fdb9da31e99389"
Nov 02 02:04:20.726621 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:04:20.726593    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6ea5b2fbb2ce852bb231bbb9131c2b5b3c248c91a8d9f6b70fdb9da31e99389\": container with ID starting with f6ea5b2fbb2ce852bb231bbb9131c2b5b3c248c91a8d9f6b70fdb9da31e99389 not found: ID does not exist" containerID="f6ea5b2fbb2ce852bb231bbb9131c2b5b3c248c91a8d9f6b70fdb9da31e99389"
Nov 02 02:04:20.726716 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.726628    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6ea5b2fbb2ce852bb231bbb9131c2b5b3c248c91a8d9f6b70fdb9da31e99389"} err="failed to get container status \"f6ea5b2fbb2ce852bb231bbb9131c2b5b3c248c91a8d9f6b70fdb9da31e99389\": rpc error: code = NotFound desc = could not find container \"f6ea5b2fbb2ce852bb231bbb9131c2b5b3c248c91a8d9f6b70fdb9da31e99389\": container with ID starting with f6ea5b2fbb2ce852bb231bbb9131c2b5b3c248c91a8d9f6b70fdb9da31e99389 not found: ID does not exist"
Nov 02 02:04:20.726716 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.726645    5639 scope.go:122] "RemoveContainer" containerID="20ad0b7f82df36994903072d915deaef626076d57ad891ac85882d0b6eecebce"
Nov 02 02:04:20.737122 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.737096    5639 scope.go:122] "RemoveContainer" containerID="20ad0b7f82df36994903072d915deaef626076d57ad891ac85882d0b6eecebce"
Nov 02 02:04:20.737482 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:04:20.737461    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20ad0b7f82df36994903072d915deaef626076d57ad891ac85882d0b6eecebce\": container with ID starting with 20ad0b7f82df36994903072d915deaef626076d57ad891ac85882d0b6eecebce not found: ID does not exist" containerID="20ad0b7f82df36994903072d915deaef626076d57ad891ac85882d0b6eecebce"
Nov 02 02:04:20.737544 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.737492    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20ad0b7f82df36994903072d915deaef626076d57ad891ac85882d0b6eecebce"} err="failed to get container status \"20ad0b7f82df36994903072d915deaef626076d57ad891ac85882d0b6eecebce\": rpc error: code = NotFound desc = could not find container \"20ad0b7f82df36994903072d915deaef626076d57ad891ac85882d0b6eecebce\": container with ID starting with 20ad0b7f82df36994903072d915deaef626076d57ad891ac85882d0b6eecebce not found: ID does not exist"
Nov 02 02:04:20.879725 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.879693    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf"]
Nov 02 02:04:20.884536 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.884502    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-07ac70d0-6c86cf6ff9-zl2tf"]
Nov 02 02:04:20.994486 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.994392    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4"]
Nov 02 02:04:20.998422 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:20.998385    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-07ac70d0-599ddfc99-5tfc4"]
Nov 02 02:04:21.219977 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:21.219943    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="322c165c-ab83-4cc0-91f2-d83533aa6472" path="/var/lib/kubelet/pods/322c165c-ab83-4cc0-91f2-d83533aa6472/volumes"
Nov 02 02:04:21.220440 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:21.220425    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91fe93cf-734d-4a19-868d-b4c38a76bc2b" path="/var/lib/kubelet/pods/91fe93cf-734d-4a19-868d-b4c38a76bc2b/volumes"
Nov 02 02:04:23.249150 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:23.249122    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2"
Nov 02 02:04:23.406208 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:23.406173    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2ed806-9940-4872-93b2-0cd86f5aeb52-kube-api-access-dwvbv\" (UniqueName: \"kubernetes.io/projected/2d2ed806-9940-4872-93b2-0cd86f5aeb52-kube-api-access-dwvbv\") pod \"2d2ed806-9940-4872-93b2-0cd86f5aeb52\" (UID: \"2d2ed806-9940-4872-93b2-0cd86f5aeb52\") "
Nov 02 02:04:23.406484 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:23.406386    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d2ed806-9940-4872-93b2-0cd86f5aeb52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2ed806-9940-4872-93b2-0cd86f5aeb52-plugins-config-volume\") pod \"2d2ed806-9940-4872-93b2-0cd86f5aeb52\" (UID: \"2d2ed806-9940-4872-93b2-0cd86f5aeb52\") "
Nov 02 02:04:23.406777 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:23.406750    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2ed806-9940-4872-93b2-0cd86f5aeb52-plugins-config-volume" pod "2d2ed806-9940-4872-93b2-0cd86f5aeb52" (UID: "2d2ed806-9940-4872-93b2-0cd86f5aeb52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:04:23.408634 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:23.408610    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2ed806-9940-4872-93b2-0cd86f5aeb52-kube-api-access-dwvbv" pod "2d2ed806-9940-4872-93b2-0cd86f5aeb52" (UID: "2d2ed806-9940-4872-93b2-0cd86f5aeb52"). InnerVolumeSpecName "kube-api-access-dwvbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:04:23.508009 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:23.507930    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwvbv\" (UniqueName: \"kubernetes.io/projected/2d2ed806-9940-4872-93b2-0cd86f5aeb52-kube-api-access-dwvbv\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:23.508009 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:23.507961    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2ed806-9940-4872-93b2-0cd86f5aeb52-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:23.741677 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:23.741635    5639 generic.go:356] "Generic (PLEG): container finished" podID="2d2ed806-9940-4872-93b2-0cd86f5aeb52" containerID="91d58044a9e6dc09bfb24c95c48c3b1a073ca0ac0b669fe87a092960fcca328f" exitCode=0
Nov 02 02:04:23.741888 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:23.741699    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2"
Nov 02 02:04:23.741888 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:23.741764    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2" event={"ID":"2d2ed806-9940-4872-93b2-0cd86f5aeb52","Type":"ContainerDied","Data":"91d58044a9e6dc09bfb24c95c48c3b1a073ca0ac0b669fe87a092960fcca328f"}
Nov 02 02:04:23.741888 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:23.741860    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2" event={"ID":"2d2ed806-9940-4872-93b2-0cd86f5aeb52","Type":"ContainerDied","Data":"b035945b06e93855d72451e7a360dc41f611c470391c170d1ba0c66fbab27da3"}
Nov 02 02:04:23.742069 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:23.741890    5639 scope.go:122] "RemoveContainer" containerID="91d58044a9e6dc09bfb24c95c48c3b1a073ca0ac0b669fe87a092960fcca328f"
Nov 02 02:04:23.753482 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:23.753460    5639 scope.go:122] "RemoveContainer" containerID="91d58044a9e6dc09bfb24c95c48c3b1a073ca0ac0b669fe87a092960fcca328f"
Nov 02 02:04:23.753777 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:04:23.753746    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d58044a9e6dc09bfb24c95c48c3b1a073ca0ac0b669fe87a092960fcca328f\": container with ID starting with 91d58044a9e6dc09bfb24c95c48c3b1a073ca0ac0b669fe87a092960fcca328f not found: ID does not exist" containerID="91d58044a9e6dc09bfb24c95c48c3b1a073ca0ac0b669fe87a092960fcca328f"
Nov 02 02:04:23.753882 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:23.753789    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d58044a9e6dc09bfb24c95c48c3b1a073ca0ac0b669fe87a092960fcca328f"} err="failed to get container status \"91d58044a9e6dc09bfb24c95c48c3b1a073ca0ac0b669fe87a092960fcca328f\": rpc error: code = NotFound desc = could not find container \"91d58044a9e6dc09bfb24c95c48c3b1a073ca0ac0b669fe87a092960fcca328f\": container with ID starting with 91d58044a9e6dc09bfb24c95c48c3b1a073ca0ac0b669fe87a092960fcca328f not found: ID does not exist"
Nov 02 02:04:23.908956 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:23.908922    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2"]
Nov 02 02:04:23.911889 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:23.911863    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f276c3-77f7dd6fb7-jmqn2"]
Nov 02 02:04:24.250393 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:24.242220    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9"]
Nov 02 02:04:24.257516 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:24.257460    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-187478-nj5l9"]
Nov 02 02:04:25.245362 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:25.245327    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d2ed806-9940-4872-93b2-0cd86f5aeb52" path="/var/lib/kubelet/pods/2d2ed806-9940-4872-93b2-0cd86f5aeb52/volumes"
Nov 02 02:04:25.246022 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:25.246000    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad1ef4a4-1194-4dc5-bd7c-836717fda66e" path="/var/lib/kubelet/pods/ad1ef4a4-1194-4dc5-bd7c-836717fda66e/volumes"
Nov 02 02:04:26.236949 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:26.236910    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"]
Nov 02 02:04:26.237484 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:26.237196    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp" podUID="eb685866-3bb8-43ae-85f9-17cea849a9f8" containerName="istio-proxy" containerID="cri-o://acd0899b4ff5fccb1881344f98a75bbcac40f58045b5de9b6edec43db513774d" gracePeriod=30
Nov 02 02:04:27.423417 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:04:27.423374    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:04:31.497909 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.497880    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 02:04:31.569251 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.569152    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb685866-3bb8-43ae-85f9-17cea849a9f8-kube-api-access-2m7tk\" (UniqueName: \"kubernetes.io/projected/eb685866-3bb8-43ae-85f9-17cea849a9f8-kube-api-access-2m7tk\") pod \"eb685866-3bb8-43ae-85f9-17cea849a9f8\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") "
Nov 02 02:04:31.569251 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.569218    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb685866-3bb8-43ae-85f9-17cea849a9f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb685866-3bb8-43ae-85f9-17cea849a9f8-istiod-ca-cert\") pod \"eb685866-3bb8-43ae-85f9-17cea849a9f8\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") "
Nov 02 02:04:31.569531 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.569256    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-envoy\") pod \"eb685866-3bb8-43ae-85f9-17cea849a9f8\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") "
Nov 02 02:04:31.569531 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.569313    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-credential-socket\") pod \"eb685866-3bb8-43ae-85f9-17cea849a9f8\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") "
Nov 02 02:04:31.569531 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.569331    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-token\" (UniqueName: \"kubernetes.io/projected/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-token\") pod \"eb685866-3bb8-43ae-85f9-17cea849a9f8\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") "
Nov 02 02:04:31.569531 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.569385    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-workload-socket\") pod \"eb685866-3bb8-43ae-85f9-17cea849a9f8\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") "
Nov 02 02:04:31.569531 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.569432    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-podinfo\") pod \"eb685866-3bb8-43ae-85f9-17cea849a9f8\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") "
Nov 02 02:04:31.569802 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.569566    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-data\") pod \"eb685866-3bb8-43ae-85f9-17cea849a9f8\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") "
Nov 02 02:04:31.569802 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.569599    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-workload-certs\") pod \"eb685866-3bb8-43ae-85f9-17cea849a9f8\" (UID: \"eb685866-3bb8-43ae-85f9-17cea849a9f8\") "
Nov 02 02:04:31.569911 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.569846    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-workload-socket" pod "eb685866-3bb8-43ae-85f9-17cea849a9f8" (UID: "eb685866-3bb8-43ae-85f9-17cea849a9f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:31.569911 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.569875    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb685866-3bb8-43ae-85f9-17cea849a9f8-istiod-ca-cert" pod "eb685866-3bb8-43ae-85f9-17cea849a9f8" (UID: "eb685866-3bb8-43ae-85f9-17cea849a9f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:04:31.570197 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.570026    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-workload-certs" pod "eb685866-3bb8-43ae-85f9-17cea849a9f8" (UID: "eb685866-3bb8-43ae-85f9-17cea849a9f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:31.570197 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.570070    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-data" pod "eb685866-3bb8-43ae-85f9-17cea849a9f8" (UID: "eb685866-3bb8-43ae-85f9-17cea849a9f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:31.570197 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.570161    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-credential-socket" pod "eb685866-3bb8-43ae-85f9-17cea849a9f8" (UID: "eb685866-3bb8-43ae-85f9-17cea849a9f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:31.570403 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.570255    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:31.570403 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.570266    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:31.570403 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.570295    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb685866-3bb8-43ae-85f9-17cea849a9f8-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:31.570403 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.570309    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:31.570403 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.570319    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:31.571900 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.571869    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb685866-3bb8-43ae-85f9-17cea849a9f8-kube-api-access-2m7tk" pod "eb685866-3bb8-43ae-85f9-17cea849a9f8" (UID: "eb685866-3bb8-43ae-85f9-17cea849a9f8"). InnerVolumeSpecName "kube-api-access-2m7tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:04:31.572244 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.572224    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-envoy" pod "eb685866-3bb8-43ae-85f9-17cea849a9f8" (UID: "eb685866-3bb8-43ae-85f9-17cea849a9f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:31.572411 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.572392    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-token" pod "eb685866-3bb8-43ae-85f9-17cea849a9f8" (UID: "eb685866-3bb8-43ae-85f9-17cea849a9f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:04:31.572463 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.572414    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-podinfo" pod "eb685866-3bb8-43ae-85f9-17cea849a9f8" (UID: "eb685866-3bb8-43ae-85f9-17cea849a9f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:04:31.671944 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.671893    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m7tk\" (UniqueName: \"kubernetes.io/projected/eb685866-3bb8-43ae-85f9-17cea849a9f8-kube-api-access-2m7tk\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:31.671944 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.671932    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:31.671944 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.671944    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:31.671944 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.671952    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb685866-3bb8-43ae-85f9-17cea849a9f8-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:31.833052 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.833013    5639 generic.go:356] "Generic (PLEG): container finished" podID="eb685866-3bb8-43ae-85f9-17cea849a9f8" containerID="acd0899b4ff5fccb1881344f98a75bbcac40f58045b5de9b6edec43db513774d" exitCode=0
Nov 02 02:04:31.833272 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.833079    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"
Nov 02 02:04:31.833272 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.833106    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp" event={"ID":"eb685866-3bb8-43ae-85f9-17cea849a9f8","Type":"ContainerDied","Data":"acd0899b4ff5fccb1881344f98a75bbcac40f58045b5de9b6edec43db513774d"}
Nov 02 02:04:31.833272 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.833174    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp" event={"ID":"eb685866-3bb8-43ae-85f9-17cea849a9f8","Type":"ContainerDied","Data":"769efe5c877019b84790972fb1e060ec4e968a1c37963afd6271295d71206624"}
Nov 02 02:04:31.833272 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.833205    5639 scope.go:122] "RemoveContainer" containerID="acd0899b4ff5fccb1881344f98a75bbcac40f58045b5de9b6edec43db513774d"
Nov 02 02:04:31.845564 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.845542    5639 scope.go:122] "RemoveContainer" containerID="8d940a2ad7728f0ce218a8285737b15ad93fdda1762f17cfa23c346b8ade176c"
Nov 02 02:04:31.858752 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.858725    5639 scope.go:122] "RemoveContainer" containerID="acd0899b4ff5fccb1881344f98a75bbcac40f58045b5de9b6edec43db513774d"
Nov 02 02:04:31.859095 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:04:31.859069    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd0899b4ff5fccb1881344f98a75bbcac40f58045b5de9b6edec43db513774d\": container with ID starting with acd0899b4ff5fccb1881344f98a75bbcac40f58045b5de9b6edec43db513774d not found: ID does not exist" containerID="acd0899b4ff5fccb1881344f98a75bbcac40f58045b5de9b6edec43db513774d"
Nov 02 02:04:31.859176 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.859109    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd0899b4ff5fccb1881344f98a75bbcac40f58045b5de9b6edec43db513774d"} err="failed to get container status \"acd0899b4ff5fccb1881344f98a75bbcac40f58045b5de9b6edec43db513774d\": rpc error: code = NotFound desc = could not find container \"acd0899b4ff5fccb1881344f98a75bbcac40f58045b5de9b6edec43db513774d\": container with ID starting with acd0899b4ff5fccb1881344f98a75bbcac40f58045b5de9b6edec43db513774d not found: ID does not exist"
Nov 02 02:04:31.859176 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.859133    5639 scope.go:122] "RemoveContainer" containerID="8d940a2ad7728f0ce218a8285737b15ad93fdda1762f17cfa23c346b8ade176c"
Nov 02 02:04:31.859532 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:04:31.859507    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d940a2ad7728f0ce218a8285737b15ad93fdda1762f17cfa23c346b8ade176c\": container with ID starting with 8d940a2ad7728f0ce218a8285737b15ad93fdda1762f17cfa23c346b8ade176c not found: ID does not exist" containerID="8d940a2ad7728f0ce218a8285737b15ad93fdda1762f17cfa23c346b8ade176c"
Nov 02 02:04:31.859583 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.859540    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d940a2ad7728f0ce218a8285737b15ad93fdda1762f17cfa23c346b8ade176c"} err="failed to get container status \"8d940a2ad7728f0ce218a8285737b15ad93fdda1762f17cfa23c346b8ade176c\": rpc error: code = NotFound desc = could not find container \"8d940a2ad7728f0ce218a8285737b15ad93fdda1762f17cfa23c346b8ade176c\": container with ID starting with 8d940a2ad7728f0ce218a8285737b15ad93fdda1762f17cfa23c346b8ade176c not found: ID does not exist"
Nov 02 02:04:32.000007 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:31.999968    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"]
Nov 02 02:04:32.002825 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:32.002793    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-4slxp"]
Nov 02 02:04:33.219295 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:33.219247    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb685866-3bb8-43ae-85f9-17cea849a9f8" path="/var/lib/kubelet/pods/eb685866-3bb8-43ae-85f9-17cea849a9f8/volumes"
Nov 02 02:04:34.857229 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:34.857197    5639 generic.go:356] "Generic (PLEG): container finished" podID="dae611da-8694-4348-9dff-90bd553e95ec" containerID="a942d64662474c14cd716d120f3985690aec6ec7b5411be3468afc15aa06877e" exitCode=0
Nov 02 02:04:34.857733 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:34.857267    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd" event={"ID":"dae611da-8694-4348-9dff-90bd553e95ec","Type":"ContainerDied","Data":"a942d64662474c14cd716d120f3985690aec6ec7b5411be3468afc15aa06877e"}
Nov 02 02:04:34.857783 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:34.857754    5639 scope.go:122] "RemoveContainer" containerID="a942d64662474c14cd716d120f3985690aec6ec7b5411be3468afc15aa06877e"
Nov 02 02:04:34.859265 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:34.859235    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:04:34.859354 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:34.859332    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:04:34.859393 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:34.859367    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:04:35.889808 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:35.889770    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd" event={"ID":"dae611da-8694-4348-9dff-90bd553e95ec","Type":"ContainerStarted","Data":"49a2dc519003988e86ea98ed979c9627e2cf982073fe8a6a1e10fc5af6ccc46f"}
Nov 02 02:04:36.503191 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:36.498198    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:04:36.503191 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:36.498252    5639 patch_prober.go:29] interesting pod/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 02 02:04:36.503191 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:36.498327    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd" podUID="dae611da-8694-4348-9dff-90bd553e95ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 02 02:04:37.498372 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:37.498333    5639 patch_prober.go:29] interesting pod/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 02 02:04:37.498785 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:37.498397    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd" podUID="dae611da-8694-4348-9dff-90bd553e95ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 02 02:04:38.498484 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:38.498440    5639 patch_prober.go:29] interesting pod/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 02 02:04:38.498906 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:38.498521    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd" podUID="dae611da-8694-4348-9dff-90bd553e95ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 02 02:04:39.497813 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:39.497776    5639 patch_prober.go:29] interesting pod/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 02 02:04:39.498032 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:39.497844    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd" podUID="dae611da-8694-4348-9dff-90bd553e95ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 02 02:04:40.284301 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.283867    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj"]
Nov 02 02:04:40.284827 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.284773    5639 cpu_manager.go:414] "containerMap: removing container" podUID="2d2ed806-9940-4872-93b2-0cd86f5aeb52" containerName="payload-pre-processing"
Nov 02 02:04:40.284827 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.284799    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2ed806-9940-4872-93b2-0cd86f5aeb52" containerName="payload-pre-processing"
Nov 02 02:04:40.284960 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.284842    5639 cpu_manager.go:414] "containerMap: removing container" podUID="f7fc9915-885c-4a8e-97a4-298fa66ab8aa" containerName="revoke-keys"
Nov 02 02:04:40.284960 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.284849    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7fc9915-885c-4a8e-97a4-298fa66ab8aa" containerName="revoke-keys"
Nov 02 02:04:40.284960 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.284872    5639 cpu_manager.go:414] "containerMap: removing container" podUID="91fe93cf-734d-4a19-868d-b4c38a76bc2b" containerName="payload-processing"
Nov 02 02:04:40.284960 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.284880    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91fe93cf-734d-4a19-868d-b4c38a76bc2b" containerName="payload-processing"
Nov 02 02:04:40.284960 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.284894    5639 cpu_manager.go:414] "containerMap: removing container" podUID="eb685866-3bb8-43ae-85f9-17cea849a9f8" containerName="istio-proxy"
Nov 02 02:04:40.284960 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.284902    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb685866-3bb8-43ae-85f9-17cea849a9f8" containerName="istio-proxy"
Nov 02 02:04:40.284960 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.284912    5639 cpu_manager.go:414] "containerMap: removing container" podUID="eb685866-3bb8-43ae-85f9-17cea849a9f8" containerName="istio-proxy"
Nov 02 02:04:40.284960 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.284919    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb685866-3bb8-43ae-85f9-17cea849a9f8" containerName="istio-proxy"
Nov 02 02:04:40.285425 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.284968    5639 cpu_manager.go:414] "containerMap: removing container" podUID="322c165c-ab83-4cc0-91f2-d83533aa6472" containerName="payload-pre-processing"
Nov 02 02:04:40.285425 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.284975    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="322c165c-ab83-4cc0-91f2-d83533aa6472" containerName="payload-pre-processing"
Nov 02 02:04:40.285425 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.285117    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2ed806-9940-4872-93b2-0cd86f5aeb52" containerName="payload-pre-processing"
Nov 02 02:04:40.285425 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.285135    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7fc9915-885c-4a8e-97a4-298fa66ab8aa" containerName="revoke-keys"
Nov 02 02:04:40.285425 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.285149    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="91fe93cf-734d-4a19-868d-b4c38a76bc2b" containerName="payload-processing"
Nov 02 02:04:40.285425 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.285160    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb685866-3bb8-43ae-85f9-17cea849a9f8" containerName="istio-proxy"
Nov 02 02:04:40.285425 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.285168    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb685866-3bb8-43ae-85f9-17cea849a9f8" containerName="istio-proxy"
Nov 02 02:04:40.285425 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.285183    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="322c165c-ab83-4cc0-91f2-d83533aa6472" containerName="payload-pre-processing"
Nov 02 02:04:40.309308 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.297143    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj"
Nov 02 02:04:40.316131 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.316095    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj"]
Nov 02 02:04:40.404245 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.404210    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d0fdb9-dockercfg-ngvjx\""
Nov 02 02:04:40.404553 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.404390    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d0fdb9\""
Nov 02 02:04:40.435416 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.435376    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/585317a1-6f66-4c71-88a5-bb3089999ff7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj\" (UID: \"585317a1-6f66-4c71-88a5-bb3089999ff7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj"
Nov 02 02:04:40.435602 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.435534    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znndn\" (UniqueName: \"kubernetes.io/projected/585317a1-6f66-4c71-88a5-bb3089999ff7-kube-api-access-znndn\") pod \"payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj\" (UID: \"585317a1-6f66-4c71-88a5-bb3089999ff7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj"
Nov 02 02:04:40.497797 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.497758    5639 patch_prober.go:29] interesting pod/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 02 02:04:40.497980 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.497823    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd" podUID="dae611da-8694-4348-9dff-90bd553e95ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 02 02:04:40.537334 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.537220    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/585317a1-6f66-4c71-88a5-bb3089999ff7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj\" (UID: \"585317a1-6f66-4c71-88a5-bb3089999ff7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj"
Nov 02 02:04:40.537493 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.537384    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znndn\" (UniqueName: \"kubernetes.io/projected/585317a1-6f66-4c71-88a5-bb3089999ff7-kube-api-access-znndn\") pod \"payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj\" (UID: \"585317a1-6f66-4c71-88a5-bb3089999ff7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj"
Nov 02 02:04:40.537991 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.537971    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/585317a1-6f66-4c71-88a5-bb3089999ff7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj\" (UID: \"585317a1-6f66-4c71-88a5-bb3089999ff7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj"
Nov 02 02:04:40.598554 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.598519    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znndn\" (UniqueName: \"kubernetes.io/projected/585317a1-6f66-4c71-88a5-bb3089999ff7-kube-api-access-znndn\") pod \"payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj\" (UID: \"585317a1-6f66-4c71-88a5-bb3089999ff7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj"
Nov 02 02:04:40.653264 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:40.653230    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj"
Nov 02 02:04:41.059095 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:41.059060    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj"]
Nov 02 02:04:41.060256 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:04:41.060215    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod585317a1_6f66_4c71_88a5_bb3089999ff7.slice/crio-0368a3ad9fd6298dda14082cf8b1e566c4e3133fa522402878146046cd0c6a63 WatchSource:0}: Error finding container 0368a3ad9fd6298dda14082cf8b1e566c4e3133fa522402878146046cd0c6a63: Status 404 returned error can't find the container with id 0368a3ad9fd6298dda14082cf8b1e566c4e3133fa522402878146046cd0c6a63
Nov 02 02:04:41.503113 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:41.503083    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:04:41.503672 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:41.503575    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:04:41.504191 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:41.504171    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:04:41.941493 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:41.941457    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj" event={"ID":"585317a1-6f66-4c71-88a5-bb3089999ff7","Type":"ContainerStarted","Data":"69db31dd89634c73a589993b8fbe43ad52a8f92da2054780a4dc2bb228a32b8e"}
Nov 02 02:04:41.941715 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:41.941514    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj" event={"ID":"585317a1-6f66-4c71-88a5-bb3089999ff7","Type":"ContainerStarted","Data":"0368a3ad9fd6298dda14082cf8b1e566c4e3133fa522402878146046cd0c6a63"}
Nov 02 02:04:41.941789 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:41.941744    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj"
Nov 02 02:04:42.061163 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:42.061108    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj" podStartSLOduration=2.061092624 podStartE2EDuration="2.061092624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:04:42.057831484 +0000 UTC m=+2346.961513629" watchObservedRunningTime="2026-11-02 02:04:42.061092624 +0000 UTC m=+2346.964774737"
Nov 02 02:04:42.579902 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:42.579869    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:04:43.405395 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.405356    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc2929a0-jdt45"]
Nov 02 02:04:43.410047 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.410021    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2929a0-jdt45"
Nov 02 02:04:43.460052 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.460014    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-143b1c14-n5mks"]
Nov 02 02:04:43.464308 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.464265    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-143b1c14-n5mks"
Nov 02 02:04:43.480535 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.480502    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d5428f-ff79-4948-8a60-8dac2796d29f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc2929a0-jdt45\" (UID: \"c6d5428f-ff79-4948-8a60-8dac2796d29f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2929a0-jdt45"
Nov 02 02:04:43.480741 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.480543    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d5428f-ff79-4948-8a60-8dac2796d29f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc2929a0-jdt45\" (UID: \"c6d5428f-ff79-4948-8a60-8dac2796d29f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2929a0-jdt45"
Nov 02 02:04:43.480741 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.480708    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkd7d\" (UniqueName: \"kubernetes.io/projected/c6d5428f-ff79-4948-8a60-8dac2796d29f-kube-api-access-qkd7d\") pod \"maas-api-revoke-sub-dc2929a0-jdt45\" (UID: \"c6d5428f-ff79-4948-8a60-8dac2796d29f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2929a0-jdt45"
Nov 02 02:04:43.506653 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.506616    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:04:43.506824 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.506670    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:04:43.509692 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.509667    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc2929a0-jdt45"]
Nov 02 02:04:43.512803 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.512786    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:04:43.512872 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.512856    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:04:43.523045 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.523011    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-143b1c14-n5mks"]
Nov 02 02:04:43.581528 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.581493    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d5428f-ff79-4948-8a60-8dac2796d29f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc2929a0-jdt45\" (UID: \"c6d5428f-ff79-4948-8a60-8dac2796d29f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2929a0-jdt45"
Nov 02 02:04:43.581528 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.581531    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d5428f-ff79-4948-8a60-8dac2796d29f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc2929a0-jdt45\" (UID: \"c6d5428f-ff79-4948-8a60-8dac2796d29f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2929a0-jdt45"
Nov 02 02:04:43.582020 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.581571    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93f67ee-ce71-45c5-a7ec-e04378559fe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-143b1c14-n5mks\" (UID: \"a93f67ee-ce71-45c5-a7ec-e04378559fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-143b1c14-n5mks"
Nov 02 02:04:43.582020 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.581620    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkd7d\" (UniqueName: \"kubernetes.io/projected/c6d5428f-ff79-4948-8a60-8dac2796d29f-kube-api-access-qkd7d\") pod \"maas-api-revoke-sub-dc2929a0-jdt45\" (UID: \"c6d5428f-ff79-4948-8a60-8dac2796d29f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2929a0-jdt45"
Nov 02 02:04:43.582020 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.581824    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93f67ee-ce71-45c5-a7ec-e04378559fe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-143b1c14-n5mks\" (UID: \"a93f67ee-ce71-45c5-a7ec-e04378559fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-143b1c14-n5mks"
Nov 02 02:04:43.582020 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.581869    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8q82\" (UniqueName: \"kubernetes.io/projected/a93f67ee-ce71-45c5-a7ec-e04378559fe2-kube-api-access-n8q82\") pod \"maas-api-revoke-sub-143b1c14-n5mks\" (UID: \"a93f67ee-ce71-45c5-a7ec-e04378559fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-143b1c14-n5mks"
Nov 02 02:04:43.582340 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.582322    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d5428f-ff79-4948-8a60-8dac2796d29f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc2929a0-jdt45\" (UID: \"c6d5428f-ff79-4948-8a60-8dac2796d29f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2929a0-jdt45"
Nov 02 02:04:43.584292 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.584245    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d5428f-ff79-4948-8a60-8dac2796d29f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc2929a0-jdt45\" (UID: \"c6d5428f-ff79-4948-8a60-8dac2796d29f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2929a0-jdt45"
Nov 02 02:04:43.643110 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.643078    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkd7d\" (UniqueName: \"kubernetes.io/projected/c6d5428f-ff79-4948-8a60-8dac2796d29f-kube-api-access-qkd7d\") pod \"maas-api-revoke-sub-dc2929a0-jdt45\" (UID: \"c6d5428f-ff79-4948-8a60-8dac2796d29f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2929a0-jdt45"
Nov 02 02:04:43.685507 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.683685    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93f67ee-ce71-45c5-a7ec-e04378559fe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-143b1c14-n5mks\" (UID: \"a93f67ee-ce71-45c5-a7ec-e04378559fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-143b1c14-n5mks"
Nov 02 02:04:43.685507 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.684072    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93f67ee-ce71-45c5-a7ec-e04378559fe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-143b1c14-n5mks\" (UID: \"a93f67ee-ce71-45c5-a7ec-e04378559fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-143b1c14-n5mks"
Nov 02 02:04:43.685507 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.684126    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8q82\" (UniqueName: \"kubernetes.io/projected/a93f67ee-ce71-45c5-a7ec-e04378559fe2-kube-api-access-n8q82\") pod \"maas-api-revoke-sub-143b1c14-n5mks\" (UID: \"a93f67ee-ce71-45c5-a7ec-e04378559fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-143b1c14-n5mks"
Nov 02 02:04:43.685507 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.684523    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93f67ee-ce71-45c5-a7ec-e04378559fe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-143b1c14-n5mks\" (UID: \"a93f67ee-ce71-45c5-a7ec-e04378559fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-143b1c14-n5mks"
Nov 02 02:04:43.686894 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.686871    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93f67ee-ce71-45c5-a7ec-e04378559fe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-143b1c14-n5mks\" (UID: \"a93f67ee-ce71-45c5-a7ec-e04378559fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-143b1c14-n5mks"
Nov 02 02:04:43.724449 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.724413    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2929a0-jdt45"
Nov 02 02:04:43.745409 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.745379    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8q82\" (UniqueName: \"kubernetes.io/projected/a93f67ee-ce71-45c5-a7ec-e04378559fe2-kube-api-access-n8q82\") pod \"maas-api-revoke-sub-143b1c14-n5mks\" (UID: \"a93f67ee-ce71-45c5-a7ec-e04378559fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-143b1c14-n5mks"
Nov 02 02:04:43.780853 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:43.780555    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-143b1c14-n5mks"
Nov 02 02:04:44.107464 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:44.107433    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc2929a0-jdt45"]
Nov 02 02:04:44.108555 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:04:44.108524    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d5428f_ff79_4948_8a60_8dac2796d29f.slice/crio-25c7ef774d4f1b73754a487a466a16720afdd8a426643e547c248581a0be2c96 WatchSource:0}: Error finding container 25c7ef774d4f1b73754a487a466a16720afdd8a426643e547c248581a0be2c96: Status 404 returned error can't find the container with id 25c7ef774d4f1b73754a487a466a16720afdd8a426643e547c248581a0be2c96
Nov 02 02:04:44.221194 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:44.221149    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-143b1c14-n5mks"]
Nov 02 02:04:44.971437 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:44.971259    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2929a0-jdt45" event={"ID":"c6d5428f-ff79-4948-8a60-8dac2796d29f","Type":"ContainerStarted","Data":"0fb3aa2d35cd3e6cb4683d05478d8c7dc91306a549116815ba5bc788ff84a05d"}
Nov 02 02:04:44.971437 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:44.971396    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2929a0-jdt45" event={"ID":"c6d5428f-ff79-4948-8a60-8dac2796d29f","Type":"ContainerStarted","Data":"25c7ef774d4f1b73754a487a466a16720afdd8a426643e547c248581a0be2c96"}
Nov 02 02:04:44.973337 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:44.973306    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-143b1c14-n5mks" event={"ID":"a93f67ee-ce71-45c5-a7ec-e04378559fe2","Type":"ContainerStarted","Data":"b7d5785f4a52e507254c9584fcc776ed3220b623053aed565caa5a798f481e29"}
Nov 02 02:04:44.973453 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:44.973367    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-143b1c14-n5mks" event={"ID":"a93f67ee-ce71-45c5-a7ec-e04378559fe2","Type":"ContainerStarted","Data":"944fca87ac8e917bf54741842e137efea64ec2d657f67ed2942022d3a3190edd"}
Nov 02 02:04:45.089021 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:45.088958    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2929a0-jdt45" podStartSLOduration=2.08894131 podStartE2EDuration="2.08894131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:04:45.085818907 +0000 UTC m=+2349.989501070" watchObservedRunningTime="2026-11-02 02:04:45.08894131 +0000 UTC m=+2349.992623423"
Nov 02 02:04:45.202879 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:45.202774    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-143b1c14-n5mks" podStartSLOduration=2.202757578 podStartE2EDuration="2.202757578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:04:45.201550315 +0000 UTC m=+2350.105232429" watchObservedRunningTime="2026-11-02 02:04:45.202757578 +0000 UTC m=+2350.106439692"
Nov 02 02:04:45.982496 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:45.982455    5639 generic.go:356] "Generic (PLEG): container finished" podID="c6d5428f-ff79-4948-8a60-8dac2796d29f" containerID="0fb3aa2d35cd3e6cb4683d05478d8c7dc91306a549116815ba5bc788ff84a05d" exitCode=0
Nov 02 02:04:45.982952 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:45.982551    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2929a0-jdt45" event={"ID":"c6d5428f-ff79-4948-8a60-8dac2796d29f","Type":"ContainerDied","Data":"0fb3aa2d35cd3e6cb4683d05478d8c7dc91306a549116815ba5bc788ff84a05d"}
Nov 02 02:04:45.984363 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:45.984332    5639 generic.go:356] "Generic (PLEG): container finished" podID="a93f67ee-ce71-45c5-a7ec-e04378559fe2" containerID="b7d5785f4a52e507254c9584fcc776ed3220b623053aed565caa5a798f481e29" exitCode=0
Nov 02 02:04:45.984502 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:45.984364    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-143b1c14-n5mks" event={"ID":"a93f67ee-ce71-45c5-a7ec-e04378559fe2","Type":"ContainerDied","Data":"b7d5785f4a52e507254c9584fcc776ed3220b623053aed565caa5a798f481e29"}
Nov 02 02:04:47.197964 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:47.197934    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-143b1c14-n5mks"
Nov 02 02:04:47.228640 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:47.228606    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2929a0-jdt45"
Nov 02 02:04:47.335867 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:47.335839    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a93f67ee-ce71-45c5-a7ec-e04378559fe2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93f67ee-ce71-45c5-a7ec-e04378559fe2-maas-api-xks-ca\") pod \"a93f67ee-ce71-45c5-a7ec-e04378559fe2\" (UID: \"a93f67ee-ce71-45c5-a7ec-e04378559fe2\") "
Nov 02 02:04:47.336024 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:47.335906    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6d5428f-ff79-4948-8a60-8dac2796d29f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d5428f-ff79-4948-8a60-8dac2796d29f-maas-api-openshift-ca\") pod \"c6d5428f-ff79-4948-8a60-8dac2796d29f\" (UID: \"c6d5428f-ff79-4948-8a60-8dac2796d29f\") "
Nov 02 02:04:47.336024 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:47.335947    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6d5428f-ff79-4948-8a60-8dac2796d29f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d5428f-ff79-4948-8a60-8dac2796d29f-maas-api-xks-ca\") pod \"c6d5428f-ff79-4948-8a60-8dac2796d29f\" (UID: \"c6d5428f-ff79-4948-8a60-8dac2796d29f\") "
Nov 02 02:04:47.336024 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:47.335998    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d5428f-ff79-4948-8a60-8dac2796d29f-kube-api-access-qkd7d\" (UniqueName: \"kubernetes.io/projected/c6d5428f-ff79-4948-8a60-8dac2796d29f-kube-api-access-qkd7d\") pod \"c6d5428f-ff79-4948-8a60-8dac2796d29f\" (UID: \"c6d5428f-ff79-4948-8a60-8dac2796d29f\") "
Nov 02 02:04:47.336225 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:47.336048    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a93f67ee-ce71-45c5-a7ec-e04378559fe2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93f67ee-ce71-45c5-a7ec-e04378559fe2-maas-api-openshift-ca\") pod \"a93f67ee-ce71-45c5-a7ec-e04378559fe2\" (UID: \"a93f67ee-ce71-45c5-a7ec-e04378559fe2\") "
Nov 02 02:04:47.336225 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:47.336128    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a93f67ee-ce71-45c5-a7ec-e04378559fe2-kube-api-access-n8q82\" (UniqueName: \"kubernetes.io/projected/a93f67ee-ce71-45c5-a7ec-e04378559fe2-kube-api-access-n8q82\") pod \"a93f67ee-ce71-45c5-a7ec-e04378559fe2\" (UID: \"a93f67ee-ce71-45c5-a7ec-e04378559fe2\") "
Nov 02 02:04:47.336363 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:47.336316    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d5428f-ff79-4948-8a60-8dac2796d29f-maas-api-openshift-ca" pod "c6d5428f-ff79-4948-8a60-8dac2796d29f" (UID: "c6d5428f-ff79-4948-8a60-8dac2796d29f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:04:47.336843 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:47.336605    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a93f67ee-ce71-45c5-a7ec-e04378559fe2-maas-api-openshift-ca" pod "a93f67ee-ce71-45c5-a7ec-e04378559fe2" (UID: "a93f67ee-ce71-45c5-a7ec-e04378559fe2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:04:47.337004 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:47.336878    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93f67ee-ce71-45c5-a7ec-e04378559fe2-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:47.337004 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:47.336900    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d5428f-ff79-4948-8a60-8dac2796d29f-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:47.338411 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:47.338384    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6d5428f-ff79-4948-8a60-8dac2796d29f-maas-api-xks-ca" pod "c6d5428f-ff79-4948-8a60-8dac2796d29f" (UID: "c6d5428f-ff79-4948-8a60-8dac2796d29f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:04:47.338814 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:47.338794    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a93f67ee-ce71-45c5-a7ec-e04378559fe2-maas-api-xks-ca" pod "a93f67ee-ce71-45c5-a7ec-e04378559fe2" (UID: "a93f67ee-ce71-45c5-a7ec-e04378559fe2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:04:47.338936 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:47.338921    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d5428f-ff79-4948-8a60-8dac2796d29f-kube-api-access-qkd7d" pod "c6d5428f-ff79-4948-8a60-8dac2796d29f" (UID: "c6d5428f-ff79-4948-8a60-8dac2796d29f"). InnerVolumeSpecName "kube-api-access-qkd7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:04:47.339006 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:47.338940    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a93f67ee-ce71-45c5-a7ec-e04378559fe2-kube-api-access-n8q82" pod "a93f67ee-ce71-45c5-a7ec-e04378559fe2" (UID: "a93f67ee-ce71-45c5-a7ec-e04378559fe2"). InnerVolumeSpecName "kube-api-access-n8q82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:04:47.438188 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:47.438153    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d5428f-ff79-4948-8a60-8dac2796d29f-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:47.438188 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:47.438186    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkd7d\" (UniqueName: \"kubernetes.io/projected/c6d5428f-ff79-4948-8a60-8dac2796d29f-kube-api-access-qkd7d\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:47.438413 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:47.438197    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8q82\" (UniqueName: \"kubernetes.io/projected/a93f67ee-ce71-45c5-a7ec-e04378559fe2-kube-api-access-n8q82\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:47.438413 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:47.438206    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93f67ee-ce71-45c5-a7ec-e04378559fe2-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:48.027470 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:48.027442    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2929a0-jdt45"
Nov 02 02:04:48.027681 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:48.027445    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2929a0-jdt45" event={"ID":"c6d5428f-ff79-4948-8a60-8dac2796d29f","Type":"ContainerDied","Data":"25c7ef774d4f1b73754a487a466a16720afdd8a426643e547c248581a0be2c96"}
Nov 02 02:04:48.027681 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:48.027595    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25c7ef774d4f1b73754a487a466a16720afdd8a426643e547c248581a0be2c96"
Nov 02 02:04:48.029007 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:48.028984    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-143b1c14-n5mks"
Nov 02 02:04:48.029146 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:48.029015    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-143b1c14-n5mks" event={"ID":"a93f67ee-ce71-45c5-a7ec-e04378559fe2","Type":"ContainerDied","Data":"944fca87ac8e917bf54741842e137efea64ec2d657f67ed2942022d3a3190edd"}
Nov 02 02:04:48.029146 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:48.029066    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="944fca87ac8e917bf54741842e137efea64ec2d657f67ed2942022d3a3190edd"
Nov 02 02:04:50.643921 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:50.643887    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf"]
Nov 02 02:04:50.692919 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:50.692888    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07ac70d0-9jpnf"]
Nov 02 02:04:51.219144 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:51.219112    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7fc9915-885c-4a8e-97a4-298fa66ab8aa" path="/var/lib/kubelet/pods/f7fc9915-885c-4a8e-97a4-298fa66ab8aa/volumes"
Nov 02 02:04:52.974963 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:52.974936    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj"
Nov 02 02:04:55.973619 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:04:55.973586    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"]
Nov 02 02:04:57.420757 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:04:57.420713    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:05:10.997524 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:10.997465    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr" podUID="983fcbba-4b11-4119-a1da-9d340c30ca51" containerName="main" containerID="cri-o://525917f823e3a88b45bf6aabde23b8005deee742b2a7d07f3bfb6f3ab5e32bae" gracePeriod=45
Nov 02 02:05:11.232495 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:11.232459    5639 generic.go:356] "Generic (PLEG): container finished" podID="983fcbba-4b11-4119-a1da-9d340c30ca51" containerID="525917f823e3a88b45bf6aabde23b8005deee742b2a7d07f3bfb6f3ab5e32bae" exitCode=0
Nov 02 02:05:11.232687 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:11.232530    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr" event={"ID":"983fcbba-4b11-4119-a1da-9d340c30ca51","Type":"ContainerDied","Data":"525917f823e3a88b45bf6aabde23b8005deee742b2a7d07f3bfb6f3ab5e32bae"}
Nov 02 02:05:11.283653 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:11.283627    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"
Nov 02 02:05:11.395674 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:11.395638    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/983fcbba-4b11-4119-a1da-9d340c30ca51-tls-certs\" (UniqueName: \"kubernetes.io/secret/983fcbba-4b11-4119-a1da-9d340c30ca51-tls-certs\") pod \"983fcbba-4b11-4119-a1da-9d340c30ca51\" (UID: \"983fcbba-4b11-4119-a1da-9d340c30ca51\") "
Nov 02 02:05:11.395873 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:11.395779    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-model-cache\") pod \"983fcbba-4b11-4119-a1da-9d340c30ca51\" (UID: \"983fcbba-4b11-4119-a1da-9d340c30ca51\") "
Nov 02 02:05:11.395873 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:11.395864    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-dshm\" (UniqueName: \"kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-dshm\") pod \"983fcbba-4b11-4119-a1da-9d340c30ca51\" (UID: \"983fcbba-4b11-4119-a1da-9d340c30ca51\") "
Nov 02 02:05:11.396012 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:11.395952    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-home\" (UniqueName: \"kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-home\") pod \"983fcbba-4b11-4119-a1da-9d340c30ca51\" (UID: \"983fcbba-4b11-4119-a1da-9d340c30ca51\") "
Nov 02 02:05:11.396012 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:11.395999    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/983fcbba-4b11-4119-a1da-9d340c30ca51-kube-api-access-bkxgr\" (UniqueName: \"kubernetes.io/projected/983fcbba-4b11-4119-a1da-9d340c30ca51-kube-api-access-bkxgr\") pod \"983fcbba-4b11-4119-a1da-9d340c30ca51\" (UID: \"983fcbba-4b11-4119-a1da-9d340c30ca51\") "
Nov 02 02:05:11.396128 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:11.396015    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-model-cache" pod "983fcbba-4b11-4119-a1da-9d340c30ca51" (UID: "983fcbba-4b11-4119-a1da-9d340c30ca51"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:05:11.396128 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:11.396050    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-tmp-dir\") pod \"983fcbba-4b11-4119-a1da-9d340c30ca51\" (UID: \"983fcbba-4b11-4119-a1da-9d340c30ca51\") "
Nov 02 02:05:11.396236 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:11.396197    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-home" pod "983fcbba-4b11-4119-a1da-9d340c30ca51" (UID: "983fcbba-4b11-4119-a1da-9d340c30ca51"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:05:11.396544 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:11.396526    5639 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-model-cache\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:11.396544 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:11.396547    5639 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-home\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:11.396678 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:11.396452    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-tmp-dir" pod "983fcbba-4b11-4119-a1da-9d340c30ca51" (UID: "983fcbba-4b11-4119-a1da-9d340c30ca51"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:05:11.398088 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:11.398066    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/983fcbba-4b11-4119-a1da-9d340c30ca51-tls-certs" pod "983fcbba-4b11-4119-a1da-9d340c30ca51" (UID: "983fcbba-4b11-4119-a1da-9d340c30ca51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:05:11.398700 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:11.398680    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-dshm" pod "983fcbba-4b11-4119-a1da-9d340c30ca51" (UID: "983fcbba-4b11-4119-a1da-9d340c30ca51"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:05:11.398777 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:11.398693    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/983fcbba-4b11-4119-a1da-9d340c30ca51-kube-api-access-bkxgr" pod "983fcbba-4b11-4119-a1da-9d340c30ca51" (UID: "983fcbba-4b11-4119-a1da-9d340c30ca51"). InnerVolumeSpecName "kube-api-access-bkxgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:05:11.497618 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:11.497575    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkxgr\" (UniqueName: \"kubernetes.io/projected/983fcbba-4b11-4119-a1da-9d340c30ca51-kube-api-access-bkxgr\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:11.497618 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:11.497605    5639 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-tmp-dir\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:11.497618 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:11.497615    5639 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/983fcbba-4b11-4119-a1da-9d340c30ca51-tls-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:11.497618 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:11.497624    5639 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/983fcbba-4b11-4119-a1da-9d340c30ca51-dshm\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:12.287301 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:12.286773    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr" event={"ID":"983fcbba-4b11-4119-a1da-9d340c30ca51","Type":"ContainerDied","Data":"0b3eee272d3e6fea8e7f2b541624958aac49e73f6ab2dbc776fec84b25b46fff"}
Nov 02 02:05:12.287301 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:12.286835    5639 scope.go:122] "RemoveContainer" containerID="525917f823e3a88b45bf6aabde23b8005deee742b2a7d07f3bfb6f3ab5e32bae"
Nov 02 02:05:12.287301 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:12.287008    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"
Nov 02 02:05:12.472700 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:12.472665    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"]
Nov 02 02:05:12.522262 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:12.522222    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-07ac70d0/e2e-lifecycle-model-07ac70d0-kserve-59cb7b75bf-ff9sr"]
Nov 02 02:05:13.220348 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:13.220311    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="983fcbba-4b11-4119-a1da-9d340c30ca51" path="/var/lib/kubelet/pods/983fcbba-4b11-4119-a1da-9d340c30ca51/volumes"
Nov 02 02:05:17.050996 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:17.050949    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c158b671-l8lwm"]
Nov 02 02:05:17.113644 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:17.113606    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c158b671-l8lwm"]
Nov 02 02:05:17.233503 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:17.233465    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c471528a-da4f-46c6-b48d-3404eec749c0" path="/var/lib/kubelet/pods/c471528a-da4f-46c6-b48d-3404eec749c0/volumes"
Nov 02 02:05:23.382346 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:23.382265    5639 generic.go:356] "Generic (PLEG): container finished" podID="0c89cdf8-4ee0-444d-a8a3-8371f375c432" containerID="5f1ba773868853577afca0f3215588dbd1a7d3afe9196e4c74e77b36ddd6b058" exitCode=0
Nov 02 02:05:23.382740 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:23.382380    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5" event={"ID":"0c89cdf8-4ee0-444d-a8a3-8371f375c432","Type":"ContainerDied","Data":"5f1ba773868853577afca0f3215588dbd1a7d3afe9196e4c74e77b36ddd6b058"}
Nov 02 02:05:23.382854 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:23.382837    5639 scope.go:122] "RemoveContainer" containerID="5f1ba773868853577afca0f3215588dbd1a7d3afe9196e4c74e77b36ddd6b058"
Nov 02 02:05:23.384624 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:23.384593    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:05:23.384721 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:23.384665    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:05:23.384721 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:23.384703    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:05:24.295398 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:24.295363    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:05:24.392353 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:24.392312    5639 generic.go:356] "Generic (PLEG): container finished" podID="7b64c8f8-821f-4d7b-a684-40433fb5396c" containerID="ecf06b1e5351a646431263dc195d597b322c9baee00af15d3f8288fa6b82f872" exitCode=0
Nov 02 02:05:24.392805 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:24.392383    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w" event={"ID":"7b64c8f8-821f-4d7b-a684-40433fb5396c","Type":"ContainerDied","Data":"ecf06b1e5351a646431263dc195d597b322c9baee00af15d3f8288fa6b82f872"}
Nov 02 02:05:24.392945 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:24.392927    5639 scope.go:122] "RemoveContainer" containerID="ecf06b1e5351a646431263dc195d597b322c9baee00af15d3f8288fa6b82f872"
Nov 02 02:05:24.394711 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:24.394682    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:05:24.394837 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:24.394751    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:05:24.394837 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:24.394795    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:05:24.394837 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:24.394816    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5" event={"ID":"0c89cdf8-4ee0-444d-a8a3-8371f375c432","Type":"ContainerStarted","Data":"d7b9e5030228b7b7a01446d7629759ee09f38d1df8df035055f4ee3603f2101d"}
Nov 02 02:05:24.398763 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:24.398743    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:05:24.400134 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:24.400113    5639 patch_prober.go:29] interesting pod/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Nov 02 02:05:24.400235 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:24.400153    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5" podUID="0c89cdf8-4ee0-444d-a8a3-8371f375c432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Nov 02 02:05:25.406852 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:25.406814    5639 patch_prober.go:29] interesting pod/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Nov 02 02:05:25.407540 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:25.407489    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5" podUID="0c89cdf8-4ee0-444d-a8a3-8371f375c432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Nov 02 02:05:25.425577 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:25.425538    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w" event={"ID":"7b64c8f8-821f-4d7b-a684-40433fb5396c","Type":"ContainerStarted","Data":"662c817bc9544e127aa4e2cce28f109003a29057122dbf5031ede4cdf4e1f7f0"}
Nov 02 02:05:26.398754 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:26.398714    5639 patch_prober.go:29] interesting pod/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Nov 02 02:05:26.399109 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:26.399079    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5" podUID="0c89cdf8-4ee0-444d-a8a3-8371f375c432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Nov 02 02:05:26.423651 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:26.423620    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:05:26.424130 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:26.423877    5639 patch_prober.go:29] interesting pod/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Nov 02 02:05:26.424257 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:26.424153    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w" podUID="7b64c8f8-821f-4d7b-a684-40433fb5396c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Nov 02 02:05:27.402305 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:27.400852    5639 patch_prober.go:29] interesting pod/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Nov 02 02:05:27.402305 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:27.400928    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5" podUID="0c89cdf8-4ee0-444d-a8a3-8371f375c432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Nov 02 02:05:27.423576 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:27.423544    5639 patch_prober.go:29] interesting pod/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Nov 02 02:05:27.423792 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:27.423771    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w" podUID="7b64c8f8-821f-4d7b-a684-40433fb5396c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Nov 02 02:05:27.434564 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:05:27.434535    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:05:28.407303 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:28.402733    5639 patch_prober.go:29] interesting pod/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Nov 02 02:05:28.407303 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:28.402803    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5" podUID="0c89cdf8-4ee0-444d-a8a3-8371f375c432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Nov 02 02:05:28.423650 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:28.423614    5639 patch_prober.go:29] interesting pod/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Nov 02 02:05:28.423817 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:28.423666    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w" podUID="7b64c8f8-821f-4d7b-a684-40433fb5396c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Nov 02 02:05:29.401587 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:29.399704    5639 patch_prober.go:29] interesting pod/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Nov 02 02:05:29.401587 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:29.399772    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5" podUID="0c89cdf8-4ee0-444d-a8a3-8371f375c432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Nov 02 02:05:29.424296 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:29.424248    5639 patch_prober.go:29] interesting pod/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Nov 02 02:05:29.424685 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:29.424317    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w" podUID="7b64c8f8-821f-4d7b-a684-40433fb5396c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Nov 02 02:05:30.425344 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:30.424645    5639 patch_prober.go:29] interesting pod/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Nov 02 02:05:30.425344 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:30.424746    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w" podUID="7b64c8f8-821f-4d7b-a684-40433fb5396c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Nov 02 02:05:30.618935 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:30.618885    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:05:30.619487 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:30.619458    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:05:30.620200 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:30.620177    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:05:31.359601 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:31.359564    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:05:31.428406 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:31.428374    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:05:31.491752 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:31.491722    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:05:34.635507 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.635471    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"]
Nov 02 02:05:34.636069 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.636055    5639 cpu_manager.go:414] "containerMap: removing container" podUID="a93f67ee-ce71-45c5-a7ec-e04378559fe2" containerName="revoke-keys"
Nov 02 02:05:34.636069 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.636069    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a93f67ee-ce71-45c5-a7ec-e04378559fe2" containerName="revoke-keys"
Nov 02 02:05:34.636149 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.636085    5639 cpu_manager.go:414] "containerMap: removing container" podUID="c6d5428f-ff79-4948-8a60-8dac2796d29f" containerName="revoke-keys"
Nov 02 02:05:34.636149 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.636090    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d5428f-ff79-4948-8a60-8dac2796d29f" containerName="revoke-keys"
Nov 02 02:05:34.636149 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.636102    5639 cpu_manager.go:414] "containerMap: removing container" podUID="983fcbba-4b11-4119-a1da-9d340c30ca51" containerName="main"
Nov 02 02:05:34.636149 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.636108    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="983fcbba-4b11-4119-a1da-9d340c30ca51" containerName="main"
Nov 02 02:05:34.636326 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.636184    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d5428f-ff79-4948-8a60-8dac2796d29f" containerName="revoke-keys"
Nov 02 02:05:34.636326 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.636194    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="983fcbba-4b11-4119-a1da-9d340c30ca51" containerName="main"
Nov 02 02:05:34.636326 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.636206    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="a93f67ee-ce71-45c5-a7ec-e04378559fe2" containerName="revoke-keys"
Nov 02 02:05:34.638539 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.638523    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:34.713803 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.713771    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-361fcc-openshift-default-dockercfg-bflq2\""
Nov 02 02:05:34.759321 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.757185    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"]
Nov 02 02:05:34.839642 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.839610    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-token\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:34.839841 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.839719    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-credential-socket\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:34.839841 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.839769    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tb5x\" (UniqueName: \"kubernetes.io/projected/2e47ced9-b31c-4196-8b5c-0e980cf9297e-kube-api-access-2tb5x\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:34.839841 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.839811    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-data\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:34.840020 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.839965    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-workload-socket\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:34.840020 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.839994    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-envoy\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:34.840155 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.840117    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-podinfo\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:34.840244 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.840207    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istiod-ca-cert\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:34.840244 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.840236    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-workload-certs\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:34.941980 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.941901    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-workload-socket\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:34.941980 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.941938    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-envoy\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:34.942222 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.942049    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-podinfo\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:34.942222 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.942151    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istiod-ca-cert\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:34.942222 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.942183    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-workload-certs\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:34.942438 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.942319    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-token\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:34.942438 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.942378    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-credential-socket\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:34.942438 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.942417    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-workload-socket\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:34.942622 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.942419    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tb5x\" (UniqueName: \"kubernetes.io/projected/2e47ced9-b31c-4196-8b5c-0e980cf9297e-kube-api-access-2tb5x\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:34.942622 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.942481    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-data\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:34.942622 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.942508    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-workload-certs\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:34.942969 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.942898    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-data\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:34.943247 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.943205    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istiod-ca-cert\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:34.943247 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.943213    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-credential-socket\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:34.944866 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.944847    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-envoy\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:34.944966 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:34.944911    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-podinfo\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:35.004040 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:35.004004    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-token\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:35.004220 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:35.004141    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tb5x\" (UniqueName: \"kubernetes.io/projected/2e47ced9-b31c-4196-8b5c-0e980cf9297e-kube-api-access-2tb5x\") pod \"e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") " pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:35.315199 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:35.315113    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-361fcc-openshift-default-dockercfg-bflq2\""
Nov 02 02:05:35.323655 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:35.323622    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:35.758409 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:35.758381    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"]
Nov 02 02:05:35.759588 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:05:35.759552    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e47ced9_b31c_4196_8b5c_0e980cf9297e.slice/crio-cf259a5e3586db09d56518a847f0dc100342d52734f814adad1919c68e2c3b51 WatchSource:0}: Error finding container cf259a5e3586db09d56518a847f0dc100342d52734f814adad1919c68e2c3b51: Status 404 returned error can't find the container with id cf259a5e3586db09d56518a847f0dc100342d52734f814adad1919c68e2c3b51
Nov 02 02:05:35.762533 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:35.762496    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:05:35.762651 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:35.762577    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:05:35.762651 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:35.762626    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:05:36.543394 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:36.543361    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g" event={"ID":"2e47ced9-b31c-4196-8b5c-0e980cf9297e","Type":"ContainerStarted","Data":"8e11838eba77646636a18a852d2371239793b4c0c738887baa3cd07a7b902250"}
Nov 02 02:05:36.543562 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:36.543411    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g" event={"ID":"2e47ced9-b31c-4196-8b5c-0e980cf9297e","Type":"ContainerStarted","Data":"cf259a5e3586db09d56518a847f0dc100342d52734f814adad1919c68e2c3b51"}
Nov 02 02:05:36.667485 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:36.667435    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g" podStartSLOduration=2.667421154 podStartE2EDuration="2.667421154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:05:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:05:36.664513089 +0000 UTC m=+2401.568195206" watchObservedRunningTime="2026-11-02 02:05:36.667421154 +0000 UTC m=+2401.571103400"
Nov 02 02:05:36.959769 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:36.959740    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:05:37.324596 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:37.324509    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:37.331521 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:37.331497    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:37.550314 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:37.550267    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:37.551412 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:37.551391    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:05:38.332130 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:38.332090    5639 scope.go:122] "RemoveContainer" containerID="75ac57b658a4da3c5dd35512022e6788a47d8c62fe7878c64ccc42eafe3540ed"
Nov 02 02:05:43.727524 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:43.727492    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49"]
Nov 02 02:05:43.776037 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:43.775991    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f276c3-cvj49"]
Nov 02 02:05:45.218959 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:45.218926    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95f26f50-842b-4bc8-a059-e838e04ef4f8" path="/var/lib/kubelet/pods/95f26f50-842b-4bc8-a059-e838e04ef4f8/volumes"
Nov 02 02:05:51.439882 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:51.439850    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"]
Nov 02 02:05:51.440394 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:51.440043    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5" podUID="50ac256a-961a-4ff2-b454-06a201660154" containerName="istio-proxy" containerID="cri-o://20467e2fc98dc9d897326babfe7d0fa173b52a442c07a4b3bbdd338585e8d102" gracePeriod=30
Nov 02 02:05:56.689626 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.689594    5639 generic.go:356] "Generic (PLEG): container finished" podID="50ac256a-961a-4ff2-b454-06a201660154" containerID="20467e2fc98dc9d897326babfe7d0fa173b52a442c07a4b3bbdd338585e8d102" exitCode=0
Nov 02 02:05:56.690053 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.689673    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5" event={"ID":"50ac256a-961a-4ff2-b454-06a201660154","Type":"ContainerDied","Data":"20467e2fc98dc9d897326babfe7d0fa173b52a442c07a4b3bbdd338585e8d102"}
Nov 02 02:05:56.725379 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.725315    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 02:05:56.885570 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.885540    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50ac256a-961a-4ff2-b454-06a201660154-kube-api-access-gb2lq\" (UniqueName: \"kubernetes.io/projected/50ac256a-961a-4ff2-b454-06a201660154-kube-api-access-gb2lq\") pod \"50ac256a-961a-4ff2-b454-06a201660154\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") "
Nov 02 02:05:56.885768 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.885605    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-workload-certs\") pod \"50ac256a-961a-4ff2-b454-06a201660154\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") "
Nov 02 02:05:56.885768 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.885637    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-istio-data\") pod \"50ac256a-961a-4ff2-b454-06a201660154\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") "
Nov 02 02:05:56.885768 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.885743    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-credential-socket\") pod \"50ac256a-961a-4ff2-b454-06a201660154\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") "
Nov 02 02:05:56.885963 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.885856    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50ac256a-961a-4ff2-b454-06a201660154-istio-token\" (UniqueName: \"kubernetes.io/projected/50ac256a-961a-4ff2-b454-06a201660154-istio-token\") pod \"50ac256a-961a-4ff2-b454-06a201660154\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") "
Nov 02 02:05:56.885963 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.885904    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50ac256a-961a-4ff2-b454-06a201660154-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ac256a-961a-4ff2-b454-06a201660154-istiod-ca-cert\") pod \"50ac256a-961a-4ff2-b454-06a201660154\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") "
Nov 02 02:05:56.886077 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.885959    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-istio-data" pod "50ac256a-961a-4ff2-b454-06a201660154" (UID: "50ac256a-961a-4ff2-b454-06a201660154"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:05:56.886077 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.885970    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-workload-certs" pod "50ac256a-961a-4ff2-b454-06a201660154" (UID: "50ac256a-961a-4ff2-b454-06a201660154"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:05:56.886077 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.885980    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50ac256a-961a-4ff2-b454-06a201660154-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ac256a-961a-4ff2-b454-06a201660154-istio-podinfo\") pod \"50ac256a-961a-4ff2-b454-06a201660154\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") "
Nov 02 02:05:56.886077 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.886054    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-workload-socket\") pod \"50ac256a-961a-4ff2-b454-06a201660154\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") "
Nov 02 02:05:56.886333 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.886098    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-istio-envoy\") pod \"50ac256a-961a-4ff2-b454-06a201660154\" (UID: \"50ac256a-961a-4ff2-b454-06a201660154\") "
Nov 02 02:05:56.886414 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.886399    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50ac256a-961a-4ff2-b454-06a201660154-istiod-ca-cert" pod "50ac256a-961a-4ff2-b454-06a201660154" (UID: "50ac256a-961a-4ff2-b454-06a201660154"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:05:56.886482 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.886400    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-workload-socket" pod "50ac256a-961a-4ff2-b454-06a201660154" (UID: "50ac256a-961a-4ff2-b454-06a201660154"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:05:56.886642 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.886621    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-credential-socket" pod "50ac256a-961a-4ff2-b454-06a201660154" (UID: "50ac256a-961a-4ff2-b454-06a201660154"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:05:56.886755 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.886736    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:56.886816 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.886760    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:56.886816 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.886774    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:56.886816 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.886788    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:56.886816 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.886800    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ac256a-961a-4ff2-b454-06a201660154-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:56.888573 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.888545    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ac256a-961a-4ff2-b454-06a201660154-kube-api-access-gb2lq" pod "50ac256a-961a-4ff2-b454-06a201660154" (UID: "50ac256a-961a-4ff2-b454-06a201660154"). InnerVolumeSpecName "kube-api-access-gb2lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:05:56.888766 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.888746    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-istio-envoy" pod "50ac256a-961a-4ff2-b454-06a201660154" (UID: "50ac256a-961a-4ff2-b454-06a201660154"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:05:56.888958 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.888926    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50ac256a-961a-4ff2-b454-06a201660154-istio-podinfo" pod "50ac256a-961a-4ff2-b454-06a201660154" (UID: "50ac256a-961a-4ff2-b454-06a201660154"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:05:56.892445 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.892418    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ac256a-961a-4ff2-b454-06a201660154-istio-token" pod "50ac256a-961a-4ff2-b454-06a201660154" (UID: "50ac256a-961a-4ff2-b454-06a201660154"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:05:56.988148 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.988069    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50ac256a-961a-4ff2-b454-06a201660154-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:56.988148 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.988098    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ac256a-961a-4ff2-b454-06a201660154-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:56.988148 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.988108    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ac256a-961a-4ff2-b454-06a201660154-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:56.988148 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:56.988119    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb2lq\" (UniqueName: \"kubernetes.io/projected/50ac256a-961a-4ff2-b454-06a201660154-kube-api-access-gb2lq\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:57.420855 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:05:57.420809    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:05:57.698645 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:57.698562    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"
Nov 02 02:05:57.698645 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:57.698587    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5" event={"ID":"50ac256a-961a-4ff2-b454-06a201660154","Type":"ContainerDied","Data":"1766f6f1efdd556b190c4b3f433b53ff3501e6f66cf2e4147242ae45c82f502b"}
Nov 02 02:05:57.699196 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:57.698661    5639 scope.go:122] "RemoveContainer" containerID="20467e2fc98dc9d897326babfe7d0fa173b52a442c07a4b3bbdd338585e8d102"
Nov 02 02:05:57.915329 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:57.915293    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"]
Nov 02 02:05:57.964849 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:57.964769    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-wdpq5"]
Nov 02 02:05:59.219390 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:05:59.219358    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50ac256a-961a-4ff2-b454-06a201660154" path="/var/lib/kubelet/pods/50ac256a-961a-4ff2-b454-06a201660154/volumes"
Nov 02 02:06:04.607066 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:04.607026    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd"]
Nov 02 02:06:04.608208 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:04.608181    5639 cpu_manager.go:414] "containerMap: removing container" podUID="50ac256a-961a-4ff2-b454-06a201660154" containerName="istio-proxy"
Nov 02 02:06:04.608208 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:04.608206    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50ac256a-961a-4ff2-b454-06a201660154" containerName="istio-proxy"
Nov 02 02:06:04.608467 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:04.608405    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="50ac256a-961a-4ff2-b454-06a201660154" containerName="istio-proxy"
Nov 02 02:06:04.612850 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:04.612826    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd"
Nov 02 02:06:04.664002 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:04.663968    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd"]
Nov 02 02:06:04.693858 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:04.693823    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdddh\" (UniqueName: \"kubernetes.io/projected/4b5f60d5-7e21-48f5-96fb-ad1c519bfae0-kube-api-access-qdddh\") pod \"payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd\" (UID: \"4b5f60d5-7e21-48f5-96fb-ad1c519bfae0\") " pod="openshift-ingress/payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd"
Nov 02 02:06:04.694056 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:04.693935    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b5f60d5-7e21-48f5-96fb-ad1c519bfae0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd\" (UID: \"4b5f60d5-7e21-48f5-96fb-ad1c519bfae0\") " pod="openshift-ingress/payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd"
Nov 02 02:06:04.709817 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:04.709784    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c78625\""
Nov 02 02:06:04.709817 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:04.709798    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c78625-dockercfg-28cgt\""
Nov 02 02:06:04.795190 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:04.795159    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdddh\" (UniqueName: \"kubernetes.io/projected/4b5f60d5-7e21-48f5-96fb-ad1c519bfae0-kube-api-access-qdddh\") pod \"payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd\" (UID: \"4b5f60d5-7e21-48f5-96fb-ad1c519bfae0\") " pod="openshift-ingress/payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd"
Nov 02 02:06:04.795389 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:04.795229    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b5f60d5-7e21-48f5-96fb-ad1c519bfae0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd\" (UID: \"4b5f60d5-7e21-48f5-96fb-ad1c519bfae0\") " pod="openshift-ingress/payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd"
Nov 02 02:06:04.795977 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:04.795958    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b5f60d5-7e21-48f5-96fb-ad1c519bfae0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd\" (UID: \"4b5f60d5-7e21-48f5-96fb-ad1c519bfae0\") " pod="openshift-ingress/payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd"
Nov 02 02:06:04.856742 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:04.856714    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdddh\" (UniqueName: \"kubernetes.io/projected/4b5f60d5-7e21-48f5-96fb-ad1c519bfae0-kube-api-access-qdddh\") pod \"payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd\" (UID: \"4b5f60d5-7e21-48f5-96fb-ad1c519bfae0\") " pod="openshift-ingress/payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd"
Nov 02 02:06:04.933525 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:04.933448    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd"
Nov 02 02:06:05.310221 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:05.310191    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd"]
Nov 02 02:06:05.311623 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:06:05.311590    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b5f60d5_7e21_48f5_96fb_ad1c519bfae0.slice/crio-ee0cceb017fa48b5213d09ffe4bc195e04a33ee1b24e2f61716bd57c2737cc62 WatchSource:0}: Error finding container ee0cceb017fa48b5213d09ffe4bc195e04a33ee1b24e2f61716bd57c2737cc62: Status 404 returned error can't find the container with id ee0cceb017fa48b5213d09ffe4bc195e04a33ee1b24e2f61716bd57c2737cc62
Nov 02 02:06:05.759479 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:05.759437    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd" event={"ID":"4b5f60d5-7e21-48f5-96fb-ad1c519bfae0","Type":"ContainerStarted","Data":"50ebd5d05282e38bdcc5dc6efa7482f651f83226e7fb60836e96bdfa31929482"}
Nov 02 02:06:05.759979 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:05.759519    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd" event={"ID":"4b5f60d5-7e21-48f5-96fb-ad1c519bfae0","Type":"ContainerStarted","Data":"ee0cceb017fa48b5213d09ffe4bc195e04a33ee1b24e2f61716bd57c2737cc62"}
Nov 02 02:06:05.759979 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:05.759550    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd"
Nov 02 02:06:05.883897 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:05.883852    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd" podStartSLOduration=1.883838769 podStartE2EDuration="1.883838769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:06:05.882975044 +0000 UTC m=+2430.786657157" watchObservedRunningTime="2026-11-02 02:06:05.883838769 +0000 UTC m=+2430.787520883"
Nov 02 02:06:07.639054 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:07.639016    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:06:16.772152 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:16.772075    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd"
Nov 02 02:06:27.421411 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:06:27.421374    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:06:35.137792 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.137737    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"]
Nov 02 02:06:35.142107 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.142084    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.187407 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.187371    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"]
Nov 02 02:06:35.191270 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.191247    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.220080 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.220053    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.220080 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.220082    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389a930e-b478-448a-a3e6-a3466a362c22-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.220307 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.220102    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.220307 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.220179    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.220307 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.220262    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.220467 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.220308    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389a930e-b478-448a-a3e6-a3466a362c22-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.220467 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.220370    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xdpq\" (UniqueName: \"kubernetes.io/projected/6dbfefd1-0270-4d23-be4b-b4f3561e877e-kube-api-access-4xdpq\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.220467 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.220401    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzc4d\" (UniqueName: \"kubernetes.io/projected/389a930e-b478-448a-a3e6-a3466a362c22-kube-api-access-jzc4d\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.220467 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.220422    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.220467 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.220439    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.220641 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.220580    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.220641 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.220606    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.220641 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.220625    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.220760 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.220651    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.220760 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.220692    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389a930e-b478-448a-a3e6-a3466a362c22-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.220760 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.220749    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.220888 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.220765    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.220888 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.220790    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.244242 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.244209    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"]
Nov 02 02:06:35.312967 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.312932    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"]
Nov 02 02:06:35.321782 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.321754    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.322000 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.321817    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.322000 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.321842    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389a930e-b478-448a-a3e6-a3466a362c22-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.322000 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.321863    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.322000 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.321891    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.322000 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.321963    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.322370 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.322002    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389a930e-b478-448a-a3e6-a3466a362c22-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.322370 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.322077    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xdpq\" (UniqueName: \"kubernetes.io/projected/6dbfefd1-0270-4d23-be4b-b4f3561e877e-kube-api-access-4xdpq\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.322370 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.322152    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzc4d\" (UniqueName: \"kubernetes.io/projected/389a930e-b478-448a-a3e6-a3466a362c22-kube-api-access-jzc4d\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.322370 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.322190    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.322370 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.322220    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.322639 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.322402    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.322639 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.322449    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.322639 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.322509    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.322639 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.322564    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.322894 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.322871    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.322961 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.322566    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.322961 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.322939    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389a930e-b478-448a-a3e6-a3466a362c22-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.323101 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.322987    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.323101 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.323017    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.323258 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.323177    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.323979 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.323697    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.323979 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.323806    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.323979 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.323863    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.324983 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.324533    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389a930e-b478-448a-a3e6-a3466a362c22-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.325814 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.325260    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.325814 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.325367    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.325814 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.325510    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.326573 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.326450    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.328686 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.328657    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.328811 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.328758    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389a930e-b478-448a-a3e6-a3466a362c22-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.330101 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.330080    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.382824 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.382786    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389a930e-b478-448a-a3e6-a3466a362c22-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.385612 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.385582    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xdpq\" (UniqueName: \"kubernetes.io/projected/6dbfefd1-0270-4d23-be4b-b4f3561e877e-kube-api-access-4xdpq\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.385739 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.385718    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzc4d\" (UniqueName: \"kubernetes.io/projected/389a930e-b478-448a-a3e6-a3466a362c22-kube-api-access-jzc4d\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h98bh\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.385819 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.385796    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4zcnn\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.460368 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.460270    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:35.505508 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.505470    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:35.895606 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.895581    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"]
Nov 02 02:06:35.897535 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:06:35.897505    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dbfefd1_0270_4d23_be4b_b4f3561e877e.slice/crio-73c44d21972ff2c38e934d9400764448970a413b736d1c912a86afa2942e0810 WatchSource:0}: Error finding container 73c44d21972ff2c38e934d9400764448970a413b736d1c912a86afa2942e0810: Status 404 returned error can't find the container with id 73c44d21972ff2c38e934d9400764448970a413b736d1c912a86afa2942e0810
Nov 02 02:06:35.900179 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.900143    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:06:35.900323 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.900224    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:06:35.900323 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.900269    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:06:35.951360 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.951336    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"]
Nov 02 02:06:35.955722 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:06:35.955696    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389a930e_b478_448a_a3e6_a3466a362c22.slice/crio-a5398e9b87fa7799fcda4d99d214a8841f279a81f3290680cec0a49c3df5fcfb WatchSource:0}: Error finding container a5398e9b87fa7799fcda4d99d214a8841f279a81f3290680cec0a49c3df5fcfb: Status 404 returned error can't find the container with id a5398e9b87fa7799fcda4d99d214a8841f279a81f3290680cec0a49c3df5fcfb
Nov 02 02:06:35.958541 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.958506    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:06:35.958648 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.958575    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:06:35.958648 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:35.958604    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:06:36.026313 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:36.025928    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn" event={"ID":"6dbfefd1-0270-4d23-be4b-b4f3561e877e","Type":"ContainerStarted","Data":"12a4cadebcebad0565852db61fa3df34f6944343b6b7cdaf45355402f12da42e"}
Nov 02 02:06:36.026313 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:36.025987    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn" event={"ID":"6dbfefd1-0270-4d23-be4b-b4f3561e877e","Type":"ContainerStarted","Data":"73c44d21972ff2c38e934d9400764448970a413b736d1c912a86afa2942e0810"}
Nov 02 02:06:36.028366 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:36.028336    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh" event={"ID":"389a930e-b478-448a-a3e6-a3466a362c22","Type":"ContainerStarted","Data":"a5398e9b87fa7799fcda4d99d214a8841f279a81f3290680cec0a49c3df5fcfb"}
Nov 02 02:06:36.157701 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:36.151911    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn" podStartSLOduration=1.151895648 podStartE2EDuration="1.151895648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:06:36.146595241 +0000 UTC m=+2461.050277355" watchObservedRunningTime="2026-11-02 02:06:36.151895648 +0000 UTC m=+2461.055577762"
Nov 02 02:06:36.461105 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:36.461023    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:36.462392 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:36.462355    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 02 02:06:36.462525 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:36.462428    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn" podUID="6dbfefd1-0270-4d23-be4b-b4f3561e877e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 02 02:06:37.037929 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:37.037892    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh" event={"ID":"389a930e-b478-448a-a3e6-a3466a362c22","Type":"ContainerStarted","Data":"b0ed4f58003e70fb22a6d62908ee68a7cc3dac080f1655cda0823518dc4e5521"}
Nov 02 02:06:37.182383 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:37.172933    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh" podStartSLOduration=2.172910125 podStartE2EDuration="2.172910125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:06:37.169807397 +0000 UTC m=+2462.073489512" watchObservedRunningTime="2026-11-02 02:06:37.172910125 +0000 UTC m=+2462.076592240"
Nov 02 02:06:37.477333 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:37.474747    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 02 02:06:37.477333 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:37.474812    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn" podUID="6dbfefd1-0270-4d23-be4b-b4f3561e877e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 02 02:06:37.508309 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:37.507660    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:37.509839 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:37.509800    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-h98bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 02 02:06:37.510012 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:37.509862    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh" podUID="389a930e-b478-448a-a3e6-a3466a362c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 02 02:06:38.392307 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:38.388404    5639 scope.go:122] "RemoveContainer" containerID="c6e00145db7ebfd6af23e0e7ab221f7055ead2812f5968fff1a1b1e915b77aea"
Nov 02 02:06:38.420679 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:38.420652    5639 scope.go:122] "RemoveContainer" containerID="e4a06c6ff694e8d3220fc992ea5cfeb98196063b456d942ec18cc5a1e071f0f8"
Nov 02 02:06:38.430570 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:38.430551    5639 scope.go:122] "RemoveContainer" containerID="937743b19555396824c40b60c446a181816440ec7214fef467d1e901274e2da9"
Nov 02 02:06:38.475304 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:38.472653    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 02 02:06:38.475304 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:38.472713    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn" podUID="6dbfefd1-0270-4d23-be4b-b4f3561e877e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 02 02:06:38.512504 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:38.511654    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-h98bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 02 02:06:38.512504 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:38.511695    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh" podUID="389a930e-b478-448a-a3e6-a3466a362c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 02 02:06:39.467308 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:39.463681    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 02 02:06:39.467308 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:39.463749    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn" podUID="6dbfefd1-0270-4d23-be4b-b4f3561e877e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 02 02:06:39.506298 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:39.506234    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-h98bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 02 02:06:39.506598 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:39.506565    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh" podUID="389a930e-b478-448a-a3e6-a3466a362c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 02 02:06:40.468307 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:40.463682    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 02 02:06:40.468307 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:40.463768    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn" podUID="6dbfefd1-0270-4d23-be4b-b4f3561e877e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 02 02:06:40.509308 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:40.508735    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-h98bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 02 02:06:40.509308 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:40.508798    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh" podUID="389a930e-b478-448a-a3e6-a3466a362c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 02 02:06:41.463309 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:41.461682    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 02 02:06:41.463309 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:41.461748    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn" podUID="6dbfefd1-0270-4d23-be4b-b4f3561e877e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 02 02:06:41.508651 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:41.508615    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-h98bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 02 02:06:41.509259 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:41.509229    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh" podUID="389a930e-b478-448a-a3e6-a3466a362c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 02 02:06:42.223942 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:42.223909    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:06:42.461705 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:42.461667    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 02 02:06:42.461995 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:42.461962    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn" podUID="6dbfefd1-0270-4d23-be4b-b4f3561e877e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 02 02:06:42.506876 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:42.506797    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-h98bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 02 02:06:42.507021 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:42.506879    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh" podUID="389a930e-b478-448a-a3e6-a3466a362c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 02 02:06:43.338304 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:43.337504    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55"]
Nov 02 02:06:43.381303 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:43.375623    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55"
Nov 02 02:06:43.415407 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:43.415357    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55"]
Nov 02 02:06:43.454136 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:43.454101    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-361fcc\""
Nov 02 02:06:43.454437 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:43.454220    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-361fcc-dockercfg-76zf4\""
Nov 02 02:06:43.461485 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:43.461453    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 02 02:06:43.461646 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:43.461511    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn" podUID="6dbfefd1-0270-4d23-be4b-b4f3561e877e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 02 02:06:43.478478 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:43.478414    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c225c247-6591-43f8-b69f-d7c443f636be-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55\" (UID: \"c225c247-6591-43f8-b69f-d7c443f636be\") " pod="openshift-ingress/payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55"
Nov 02 02:06:43.478638 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:43.478546    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpwsj\" (UniqueName: \"kubernetes.io/projected/c225c247-6591-43f8-b69f-d7c443f636be-kube-api-access-zpwsj\") pod \"payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55\" (UID: \"c225c247-6591-43f8-b69f-d7c443f636be\") " pod="openshift-ingress/payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55"
Nov 02 02:06:43.506744 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:43.506711    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-h98bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 02 02:06:43.506905 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:43.506755    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh" podUID="389a930e-b478-448a-a3e6-a3466a362c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 02 02:06:43.579259 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:43.579223    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c225c247-6591-43f8-b69f-d7c443f636be-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55\" (UID: \"c225c247-6591-43f8-b69f-d7c443f636be\") " pod="openshift-ingress/payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55"
Nov 02 02:06:43.579487 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:43.579296    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpwsj\" (UniqueName: \"kubernetes.io/projected/c225c247-6591-43f8-b69f-d7c443f636be-kube-api-access-zpwsj\") pod \"payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55\" (UID: \"c225c247-6591-43f8-b69f-d7c443f636be\") " pod="openshift-ingress/payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55"
Nov 02 02:06:43.580012 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:43.579985    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c225c247-6591-43f8-b69f-d7c443f636be-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55\" (UID: \"c225c247-6591-43f8-b69f-d7c443f636be\") " pod="openshift-ingress/payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55"
Nov 02 02:06:43.640385 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:43.640311    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpwsj\" (UniqueName: \"kubernetes.io/projected/c225c247-6591-43f8-b69f-d7c443f636be-kube-api-access-zpwsj\") pod \"payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55\" (UID: \"c225c247-6591-43f8-b69f-d7c443f636be\") " pod="openshift-ingress/payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55"
Nov 02 02:06:43.723239 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:43.723201    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55"
Nov 02 02:06:44.108010 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:44.107972    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55"]
Nov 02 02:06:44.108158 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:06:44.107990    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc225c247_6591_43f8_b69f_d7c443f636be.slice/crio-ea7932d8da8b688cbca3b201e1eb4a0b28678b9293bd70ce1c78bd8cf6a0d7ff WatchSource:0}: Error finding container ea7932d8da8b688cbca3b201e1eb4a0b28678b9293bd70ce1c78bd8cf6a0d7ff: Status 404 returned error can't find the container with id ea7932d8da8b688cbca3b201e1eb4a0b28678b9293bd70ce1c78bd8cf6a0d7ff
Nov 02 02:06:44.180054 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:44.180024    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55" event={"ID":"c225c247-6591-43f8-b69f-d7c443f636be","Type":"ContainerStarted","Data":"ea7932d8da8b688cbca3b201e1eb4a0b28678b9293bd70ce1c78bd8cf6a0d7ff"}
Nov 02 02:06:44.519358 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:44.509812    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-h98bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 02 02:06:44.519358 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:44.509891    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh" podUID="389a930e-b478-448a-a3e6-a3466a362c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 02 02:06:45.194654 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:45.194519    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55" event={"ID":"c225c247-6591-43f8-b69f-d7c443f636be","Type":"ContainerStarted","Data":"ab4331e0d318fccf2a6a2e82cc7f5b5cbf4d6def3f28e42588cdcc08a4d68296"}
Nov 02 02:06:45.196320 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:45.196292    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55"
Nov 02 02:06:45.243740 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:45.243693    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:45.245012 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:45.244988    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:45.246486 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:45.246462    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:06:45.319660 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:45.318431    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55" podStartSLOduration=2.318407572 podStartE2EDuration="2.318407572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:06:45.31296331 +0000 UTC m=+2470.216645426" watchObservedRunningTime="2026-11-02 02:06:45.318407572 +0000 UTC m=+2470.222089815"
Nov 02 02:06:45.601820 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:45.601788    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:45.602562 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:45.602542    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:45.604720 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:45.604696    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:06:54.776214 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:54.776175    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4732f101-w6z8r"]
Nov 02 02:06:54.780688 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:54.780670    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732f101-w6z8r"
Nov 02 02:06:54.838002 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:54.837962    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4732f101-w6z8r"]
Nov 02 02:06:54.838607 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:54.838582    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4732f101-w6z8r\" (UID: \"f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732f101-w6z8r"
Nov 02 02:06:54.838740 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:54.838627    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4732f101-w6z8r\" (UID: \"f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732f101-w6z8r"
Nov 02 02:06:54.839161 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:54.838811    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t45f\" (UniqueName: \"kubernetes.io/projected/f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965-kube-api-access-4t45f\") pod \"maas-api-revoke-sub-4732f101-w6z8r\" (UID: \"f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732f101-w6z8r"
Nov 02 02:06:54.883716 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:54.883685    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:06:54.883716 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:54.883705    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:06:54.883716 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:54.883713    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:06:54.883983 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:54.883732    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:06:54.940333 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:54.940273    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4732f101-w6z8r\" (UID: \"f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732f101-w6z8r"
Nov 02 02:06:54.940504 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:54.940361    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4732f101-w6z8r\" (UID: \"f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732f101-w6z8r"
Nov 02 02:06:54.940563 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:54.940526    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t45f\" (UniqueName: \"kubernetes.io/projected/f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965-kube-api-access-4t45f\") pod \"maas-api-revoke-sub-4732f101-w6z8r\" (UID: \"f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732f101-w6z8r"
Nov 02 02:06:54.940913 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:54.940889    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4732f101-w6z8r\" (UID: \"f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732f101-w6z8r"
Nov 02 02:06:54.943039 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:54.943018    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4732f101-w6z8r\" (UID: \"f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732f101-w6z8r"
Nov 02 02:06:55.001840 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:55.001810    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t45f\" (UniqueName: \"kubernetes.io/projected/f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965-kube-api-access-4t45f\") pod \"maas-api-revoke-sub-4732f101-w6z8r\" (UID: \"f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732f101-w6z8r"
Nov 02 02:06:55.094373 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:55.094335    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732f101-w6z8r"
Nov 02 02:06:55.476566 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:55.476537    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4732f101-w6z8r"]
Nov 02 02:06:55.478835 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:06:55.478808    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf392f4c7_5ef3_4f2f_a7f8_d0ad1d90d965.slice/crio-ec98428c065d5cce8ee913bfe3ea9a598ac8e93c62a8436cebad0ab0985f0956 WatchSource:0}: Error finding container ec98428c065d5cce8ee913bfe3ea9a598ac8e93c62a8436cebad0ab0985f0956: Status 404 returned error can't find the container with id ec98428c065d5cce8ee913bfe3ea9a598ac8e93c62a8436cebad0ab0985f0956
Nov 02 02:06:56.206500 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:56.206466    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55"
Nov 02 02:06:56.275890 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:56.275851    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732f101-w6z8r" event={"ID":"f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965","Type":"ContainerStarted","Data":"dd1f93b06b8b3634454b6d3b837acb966eac000cfe561a349476331be66885f2"}
Nov 02 02:06:56.276068 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:56.275913    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732f101-w6z8r" event={"ID":"f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965","Type":"ContainerStarted","Data":"ec98428c065d5cce8ee913bfe3ea9a598ac8e93c62a8436cebad0ab0985f0956"}
Nov 02 02:06:56.449623 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:56.449562    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732f101-w6z8r" podStartSLOduration=2.449542528 podStartE2EDuration="2.449542528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:06:56.447401592 +0000 UTC m=+2481.351083709" watchObservedRunningTime="2026-11-02 02:06:56.449542528 +0000 UTC m=+2481.353224649"
Nov 02 02:06:57.283547 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:57.283505    5639 generic.go:356] "Generic (PLEG): container finished" podID="f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965" containerID="dd1f93b06b8b3634454b6d3b837acb966eac000cfe561a349476331be66885f2" exitCode=0
Nov 02 02:06:57.283940 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:57.283574    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732f101-w6z8r" event={"ID":"f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965","Type":"ContainerDied","Data":"dd1f93b06b8b3634454b6d3b837acb966eac000cfe561a349476331be66885f2"}
Nov 02 02:06:57.424077 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:06:57.424039    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:06:57.862115 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:57.862073    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66"]
Nov 02 02:06:57.866687 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:57.866655    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66"
Nov 02 02:06:57.910035 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:57.910002    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/264280bd-0869-4da4-a6d0-9d393849a85a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66\" (UID: \"264280bd-0869-4da4-a6d0-9d393849a85a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66"
Nov 02 02:06:57.910225 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:57.910208    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48dg8\" (UniqueName: \"kubernetes.io/projected/264280bd-0869-4da4-a6d0-9d393849a85a-kube-api-access-48dg8\") pod \"payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66\" (UID: \"264280bd-0869-4da4-a6d0-9d393849a85a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66"
Nov 02 02:06:57.920479 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:57.920437    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9039bf\""
Nov 02 02:06:57.920681 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:57.920631    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9039bf-dockercfg-2spbx\""
Nov 02 02:06:57.924764 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:57.924727    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66"]
Nov 02 02:06:58.061779 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:58.061735    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48dg8\" (UniqueName: \"kubernetes.io/projected/264280bd-0869-4da4-a6d0-9d393849a85a-kube-api-access-48dg8\") pod \"payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66\" (UID: \"264280bd-0869-4da4-a6d0-9d393849a85a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66"
Nov 02 02:06:58.062700 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:58.062665    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/264280bd-0869-4da4-a6d0-9d393849a85a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66\" (UID: \"264280bd-0869-4da4-a6d0-9d393849a85a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66"
Nov 02 02:06:58.063532 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:58.063482    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/264280bd-0869-4da4-a6d0-9d393849a85a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66\" (UID: \"264280bd-0869-4da4-a6d0-9d393849a85a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66"
Nov 02 02:06:58.123739 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:58.123668    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48dg8\" (UniqueName: \"kubernetes.io/projected/264280bd-0869-4da4-a6d0-9d393849a85a-kube-api-access-48dg8\") pod \"payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66\" (UID: \"264280bd-0869-4da4-a6d0-9d393849a85a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66"
Nov 02 02:06:58.182795 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:58.182763    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66"
Nov 02 02:06:58.522684 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:58.522657    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732f101-w6z8r"
Nov 02 02:06:58.590459 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:58.590384    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66"]
Nov 02 02:06:58.600219 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:58.600192    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965-kube-api-access-4t45f\" (UniqueName: \"kubernetes.io/projected/f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965-kube-api-access-4t45f\") pod \"f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965\" (UID: \"f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965\") "
Nov 02 02:06:58.600339 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:58.600239    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965-maas-api-openshift-ca\") pod \"f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965\" (UID: \"f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965\") "
Nov 02 02:06:58.600410 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:58.600397    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965-maas-api-xks-ca\") pod \"f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965\" (UID: \"f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965\") "
Nov 02 02:06:58.600659 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:58.600631    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965-maas-api-openshift-ca" pod "f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965" (UID: "f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:06:58.600904 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:58.600847    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:58.602582 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:58.602560    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965-kube-api-access-4t45f" pod "f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965" (UID: "f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965"). InnerVolumeSpecName "kube-api-access-4t45f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:58.602685 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:58.602665    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965-maas-api-xks-ca" pod "f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965" (UID: "f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:06:58.702377 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:58.702332    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:58.702377 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:58.702380    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t45f\" (UniqueName: \"kubernetes.io/projected/f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965-kube-api-access-4t45f\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:59.316143 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:59.316113    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732f101-w6z8r"
Nov 02 02:06:59.316363 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:59.316140    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732f101-w6z8r" event={"ID":"f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965","Type":"ContainerDied","Data":"ec98428c065d5cce8ee913bfe3ea9a598ac8e93c62a8436cebad0ab0985f0956"}
Nov 02 02:06:59.316363 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:59.316187    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec98428c065d5cce8ee913bfe3ea9a598ac8e93c62a8436cebad0ab0985f0956"
Nov 02 02:06:59.318162 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:59.318138    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66" event={"ID":"264280bd-0869-4da4-a6d0-9d393849a85a","Type":"ContainerStarted","Data":"59f7492ccb0bd350681eaf06e6e05664b247d4ef76fbc61568c65722add6c484"}
Nov 02 02:06:59.318313 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:59.318184    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66" event={"ID":"264280bd-0869-4da4-a6d0-9d393849a85a","Type":"ContainerStarted","Data":"c4ec928c84b74c4105738a491598815321a6c7748f948a9ff268a6d209c2d415"}
Nov 02 02:06:59.318313 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:59.318305    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66"
Nov 02 02:06:59.442454 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:06:59.442332    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66" podStartSLOduration=2.442317697 podStartE2EDuration="2.442317697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:06:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:06:59.440237548 +0000 UTC m=+2484.343919662" watchObservedRunningTime="2026-11-02 02:06:59.442317697 +0000 UTC m=+2484.345999808"
Nov 02 02:07:06.147204 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:06.147170    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf"]
Nov 02 02:07:06.148018 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:06.147998    5639 cpu_manager.go:414] "containerMap: removing container" podUID="f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965" containerName="revoke-keys"
Nov 02 02:07:06.148018 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:06.148018    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965" containerName="revoke-keys"
Nov 02 02:07:06.148173 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:06.148163    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965" containerName="revoke-keys"
Nov 02 02:07:06.152245 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:06.152225    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf"
Nov 02 02:07:06.204985 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:06.204802    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:07:06.220781 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:06.220741    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9r2q\" (UniqueName: \"kubernetes.io/projected/b1f67b18-24ec-42ea-b266-97bf205fc145-kube-api-access-z9r2q\") pod \"maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf\" (UID: \"b1f67b18-24ec-42ea-b266-97bf205fc145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf"
Nov 02 02:07:06.220964 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:06.220883    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f67b18-24ec-42ea-b266-97bf205fc145-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf\" (UID: \"b1f67b18-24ec-42ea-b266-97bf205fc145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf"
Nov 02 02:07:06.221040 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:06.221019    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f67b18-24ec-42ea-b266-97bf205fc145-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf\" (UID: \"b1f67b18-24ec-42ea-b266-97bf205fc145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf"
Nov 02 02:07:06.254424 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:06.254369    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:07:06.254603 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:06.254426    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:07:06.254603 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:06.254509    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:07:06.258115 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:06.258086    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf"]
Nov 02 02:07:06.322989 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:06.322943    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9r2q\" (UniqueName: \"kubernetes.io/projected/b1f67b18-24ec-42ea-b266-97bf205fc145-kube-api-access-z9r2q\") pod \"maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf\" (UID: \"b1f67b18-24ec-42ea-b266-97bf205fc145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf"
Nov 02 02:07:06.323226 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:06.323049    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f67b18-24ec-42ea-b266-97bf205fc145-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf\" (UID: \"b1f67b18-24ec-42ea-b266-97bf205fc145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf"
Nov 02 02:07:06.323226 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:06.323157    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f67b18-24ec-42ea-b266-97bf205fc145-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf\" (UID: \"b1f67b18-24ec-42ea-b266-97bf205fc145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf"
Nov 02 02:07:06.324156 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:06.324083    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f67b18-24ec-42ea-b266-97bf205fc145-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf\" (UID: \"b1f67b18-24ec-42ea-b266-97bf205fc145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf"
Nov 02 02:07:06.326033 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:06.326008    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f67b18-24ec-42ea-b266-97bf205fc145-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf\" (UID: \"b1f67b18-24ec-42ea-b266-97bf205fc145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf"
Nov 02 02:07:06.383864 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:06.383825    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9r2q\" (UniqueName: \"kubernetes.io/projected/b1f67b18-24ec-42ea-b266-97bf205fc145-kube-api-access-z9r2q\") pod \"maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf\" (UID: \"b1f67b18-24ec-42ea-b266-97bf205fc145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf"
Nov 02 02:07:06.470421 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:06.470317    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf"
Nov 02 02:07:06.872452 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:06.872421    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf"]
Nov 02 02:07:06.873499 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:07:06.873475    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f67b18_24ec_42ea_b266_97bf205fc145.slice/crio-2ec6ef2ce8bb455d31c954b120247130036048449af88554fa825d3ab846ff0f WatchSource:0}: Error finding container 2ec6ef2ce8bb455d31c954b120247130036048449af88554fa825d3ab846ff0f: Status 404 returned error can't find the container with id 2ec6ef2ce8bb455d31c954b120247130036048449af88554fa825d3ab846ff0f
Nov 02 02:07:07.391579 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:07.391540    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf" event={"ID":"b1f67b18-24ec-42ea-b266-97bf205fc145","Type":"ContainerStarted","Data":"dc2eb100db07c74aeaf9b624c9be3ef508d8b8dd26b870e4965dfb958b17f989"}
Nov 02 02:07:07.392427 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:07.391596    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf" event={"ID":"b1f67b18-24ec-42ea-b266-97bf205fc145","Type":"ContainerStarted","Data":"2ec6ef2ce8bb455d31c954b120247130036048449af88554fa825d3ab846ff0f"}
Nov 02 02:07:07.510140 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:07.508513    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf" podStartSLOduration=1.508498208 podStartE2EDuration="1.508498208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:07:07.506190648 +0000 UTC m=+2492.409872785" watchObservedRunningTime="2026-11-02 02:07:07.508498208 +0000 UTC m=+2492.412180363"
Nov 02 02:07:07.551916 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:07.551889    5639 patch_prober.go:29] interesting pod/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Nov 02 02:07:07.552022 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:07.551953    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g" podUID="2e47ced9-b31c-4196-8b5c-0e980cf9297e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Nov 02 02:07:08.400524 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:08.400488    5639 generic.go:356] "Generic (PLEG): container finished" podID="b1f67b18-24ec-42ea-b266-97bf205fc145" containerID="dc2eb100db07c74aeaf9b624c9be3ef508d8b8dd26b870e4965dfb958b17f989" exitCode=0
Nov 02 02:07:08.400957 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:08.400575    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf" event={"ID":"b1f67b18-24ec-42ea-b266-97bf205fc145","Type":"ContainerDied","Data":"dc2eb100db07c74aeaf9b624c9be3ef508d8b8dd26b870e4965dfb958b17f989"}
Nov 02 02:07:08.402209 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:08.402186    5639 generic.go:356] "Generic (PLEG): container finished" podID="2e47ced9-b31c-4196-8b5c-0e980cf9297e" containerID="8e11838eba77646636a18a852d2371239793b4c0c738887baa3cd07a7b902250" exitCode=0
Nov 02 02:07:08.402330 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:08.402258    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g" event={"ID":"2e47ced9-b31c-4196-8b5c-0e980cf9297e","Type":"ContainerDied","Data":"8e11838eba77646636a18a852d2371239793b4c0c738887baa3cd07a7b902250"}
Nov 02 02:07:08.402683 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:08.402669    5639 scope.go:122] "RemoveContainer" containerID="8e11838eba77646636a18a852d2371239793b4c0c738887baa3cd07a7b902250"
Nov 02 02:07:08.404202 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:08.404175    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:07:08.404267 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:08.404246    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:07:08.404320 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:08.404297    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:07:09.411924 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:09.411892    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g" event={"ID":"2e47ced9-b31c-4196-8b5c-0e980cf9297e","Type":"ContainerStarted","Data":"20a654a78b08fb51593eeeedee995b4b0fda68ad9d06afd08fb0e693ed2bacf1"}
Nov 02 02:07:09.616071 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:09.616044    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf"
Nov 02 02:07:09.796355 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:09.796241    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1f67b18-24ec-42ea-b266-97bf205fc145-kube-api-access-z9r2q\" (UniqueName: \"kubernetes.io/projected/b1f67b18-24ec-42ea-b266-97bf205fc145-kube-api-access-z9r2q\") pod \"b1f67b18-24ec-42ea-b266-97bf205fc145\" (UID: \"b1f67b18-24ec-42ea-b266-97bf205fc145\") "
Nov 02 02:07:09.796526 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:09.796398    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1f67b18-24ec-42ea-b266-97bf205fc145-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f67b18-24ec-42ea-b266-97bf205fc145-maas-api-openshift-ca\") pod \"b1f67b18-24ec-42ea-b266-97bf205fc145\" (UID: \"b1f67b18-24ec-42ea-b266-97bf205fc145\") "
Nov 02 02:07:09.796526 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:09.796498    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1f67b18-24ec-42ea-b266-97bf205fc145-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f67b18-24ec-42ea-b266-97bf205fc145-maas-api-xks-ca\") pod \"b1f67b18-24ec-42ea-b266-97bf205fc145\" (UID: \"b1f67b18-24ec-42ea-b266-97bf205fc145\") "
Nov 02 02:07:09.796816 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:09.796789    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1f67b18-24ec-42ea-b266-97bf205fc145-maas-api-openshift-ca" pod "b1f67b18-24ec-42ea-b266-97bf205fc145" (UID: "b1f67b18-24ec-42ea-b266-97bf205fc145"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:09.797262 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:09.797238    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f67b18-24ec-42ea-b266-97bf205fc145-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:09.798671 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:09.798644    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1f67b18-24ec-42ea-b266-97bf205fc145-maas-api-xks-ca" pod "b1f67b18-24ec-42ea-b266-97bf205fc145" (UID: "b1f67b18-24ec-42ea-b266-97bf205fc145"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:07:09.798766 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:09.798669    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f67b18-24ec-42ea-b266-97bf205fc145-kube-api-access-z9r2q" pod "b1f67b18-24ec-42ea-b266-97bf205fc145" (UID: "b1f67b18-24ec-42ea-b266-97bf205fc145"). InnerVolumeSpecName "kube-api-access-z9r2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:09.899089 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:09.899056    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f67b18-24ec-42ea-b266-97bf205fc145-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:09.899089 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:09.899087    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9r2q\" (UniqueName: \"kubernetes.io/projected/b1f67b18-24ec-42ea-b266-97bf205fc145-kube-api-access-z9r2q\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:10.324292 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.324255    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:07:10.324481 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.324445    5639 patch_prober.go:29] interesting pod/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Nov 02 02:07:10.324535 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.324501    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g" podUID="2e47ced9-b31c-4196-8b5c-0e980cf9297e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Nov 02 02:07:10.329138 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.329109    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66"
Nov 02 02:07:10.421582 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.421553    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf"
Nov 02 02:07:10.422011 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.421587    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf" event={"ID":"b1f67b18-24ec-42ea-b266-97bf205fc145","Type":"ContainerDied","Data":"2ec6ef2ce8bb455d31c954b120247130036048449af88554fa825d3ab846ff0f"}
Nov 02 02:07:10.422011 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.421638    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ec6ef2ce8bb455d31c954b120247130036048449af88554fa825d3ab846ff0f"
Nov 02 02:07:10.445869 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.445833    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"]
Nov 02 02:07:10.497052 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.497010    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg"]
Nov 02 02:07:10.497758 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.497740    5639 cpu_manager.go:414] "containerMap: removing container" podUID="b1f67b18-24ec-42ea-b266-97bf205fc145" containerName="revoke-keys"
Nov 02 02:07:10.497850 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.497760    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1f67b18-24ec-42ea-b266-97bf205fc145" containerName="revoke-keys"
Nov 02 02:07:10.498031 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.497921    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1f67b18-24ec-42ea-b266-97bf205fc145" containerName="revoke-keys"
Nov 02 02:07:10.501808 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.501789    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg"
Nov 02 02:07:10.554189 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.554150    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:07:10.556757 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.556718    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:07:10.556991 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.556957    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:07:10.595360 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.569505    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:07:10.598785 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.598440    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg"]
Nov 02 02:07:10.609969 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.608403    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31f57ff-3195-43ba-8158-99b5a3728300-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg\" (UID: \"a31f57ff-3195-43ba-8158-99b5a3728300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg"
Nov 02 02:07:10.609969 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.608497    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngkbq\" (UniqueName: \"kubernetes.io/projected/a31f57ff-3195-43ba-8158-99b5a3728300-kube-api-access-ngkbq\") pod \"maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg\" (UID: \"a31f57ff-3195-43ba-8158-99b5a3728300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg"
Nov 02 02:07:10.609969 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.608571    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31f57ff-3195-43ba-8158-99b5a3728300-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg\" (UID: \"a31f57ff-3195-43ba-8158-99b5a3728300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg"
Nov 02 02:07:10.710107 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.710064    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31f57ff-3195-43ba-8158-99b5a3728300-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg\" (UID: \"a31f57ff-3195-43ba-8158-99b5a3728300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg"
Nov 02 02:07:10.710317 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.710134    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngkbq\" (UniqueName: \"kubernetes.io/projected/a31f57ff-3195-43ba-8158-99b5a3728300-kube-api-access-ngkbq\") pod \"maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg\" (UID: \"a31f57ff-3195-43ba-8158-99b5a3728300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg"
Nov 02 02:07:10.710317 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.710193    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31f57ff-3195-43ba-8158-99b5a3728300-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg\" (UID: \"a31f57ff-3195-43ba-8158-99b5a3728300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg"
Nov 02 02:07:10.710850 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.710828    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31f57ff-3195-43ba-8158-99b5a3728300-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg\" (UID: \"a31f57ff-3195-43ba-8158-99b5a3728300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg"
Nov 02 02:07:10.714894 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.714869    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31f57ff-3195-43ba-8158-99b5a3728300-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg\" (UID: \"a31f57ff-3195-43ba-8158-99b5a3728300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg"
Nov 02 02:07:10.772769 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.772707    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngkbq\" (UniqueName: \"kubernetes.io/projected/a31f57ff-3195-43ba-8158-99b5a3728300-kube-api-access-ngkbq\") pod \"maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg\" (UID: \"a31f57ff-3195-43ba-8158-99b5a3728300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg"
Nov 02 02:07:10.820879 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:10.820847    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg"
Nov 02 02:07:11.229404 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:11.229378    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg"]
Nov 02 02:07:11.230207 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:07:11.230182    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31f57ff_3195_43ba_8158_99b5a3728300.slice/crio-9fa0b0e8d14f4336584971dcdfe1b60b78f7ee4efc255237df6a57b7f06bf073 WatchSource:0}: Error finding container 9fa0b0e8d14f4336584971dcdfe1b60b78f7ee4efc255237df6a57b7f06bf073: Status 404 returned error can't find the container with id 9fa0b0e8d14f4336584971dcdfe1b60b78f7ee4efc255237df6a57b7f06bf073
Nov 02 02:07:11.324662 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:11.324625    5639 patch_prober.go:29] interesting pod/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Nov 02 02:07:11.324840 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:11.324689    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g" podUID="2e47ced9-b31c-4196-8b5c-0e980cf9297e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Nov 02 02:07:11.429918 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:11.429887    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg" event={"ID":"a31f57ff-3195-43ba-8158-99b5a3728300","Type":"ContainerStarted","Data":"9fa0b0e8d14f4336584971dcdfe1b60b78f7ee4efc255237df6a57b7f06bf073"}
Nov 02 02:07:12.324560 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:12.324524    5639 patch_prober.go:29] interesting pod/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Nov 02 02:07:12.324740 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:12.324582    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g" podUID="2e47ced9-b31c-4196-8b5c-0e980cf9297e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Nov 02 02:07:12.438846 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:12.438817    5639 generic.go:356] "Generic (PLEG): container finished" podID="a31f57ff-3195-43ba-8158-99b5a3728300" containerID="023045ad613c6121fbbbd5069a6677a2a60dc61df52229c56833c0b58238607c" exitCode=0
Nov 02 02:07:12.439196 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:12.438897    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg" event={"ID":"a31f57ff-3195-43ba-8158-99b5a3728300","Type":"ContainerDied","Data":"023045ad613c6121fbbbd5069a6677a2a60dc61df52229c56833c0b58238607c"}
Nov 02 02:07:13.324598 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:13.324499    5639 patch_prober.go:29] interesting pod/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Nov 02 02:07:13.324598 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:13.324554    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g" podUID="2e47ced9-b31c-4196-8b5c-0e980cf9297e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Nov 02 02:07:13.628432 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:13.628402    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg"
Nov 02 02:07:13.654890 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:13.654862    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a31f57ff-3195-43ba-8158-99b5a3728300-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31f57ff-3195-43ba-8158-99b5a3728300-maas-api-xks-ca\") pod \"a31f57ff-3195-43ba-8158-99b5a3728300\" (UID: \"a31f57ff-3195-43ba-8158-99b5a3728300\") "
Nov 02 02:07:13.655068 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:13.654922    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a31f57ff-3195-43ba-8158-99b5a3728300-kube-api-access-ngkbq\" (UniqueName: \"kubernetes.io/projected/a31f57ff-3195-43ba-8158-99b5a3728300-kube-api-access-ngkbq\") pod \"a31f57ff-3195-43ba-8158-99b5a3728300\" (UID: \"a31f57ff-3195-43ba-8158-99b5a3728300\") "
Nov 02 02:07:13.655068 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:13.654991    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a31f57ff-3195-43ba-8158-99b5a3728300-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31f57ff-3195-43ba-8158-99b5a3728300-maas-api-openshift-ca\") pod \"a31f57ff-3195-43ba-8158-99b5a3728300\" (UID: \"a31f57ff-3195-43ba-8158-99b5a3728300\") "
Nov 02 02:07:13.655438 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:13.655413    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a31f57ff-3195-43ba-8158-99b5a3728300-maas-api-openshift-ca" pod "a31f57ff-3195-43ba-8158-99b5a3728300" (UID: "a31f57ff-3195-43ba-8158-99b5a3728300"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:13.657319 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:13.657298    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a31f57ff-3195-43ba-8158-99b5a3728300-maas-api-xks-ca" pod "a31f57ff-3195-43ba-8158-99b5a3728300" (UID: "a31f57ff-3195-43ba-8158-99b5a3728300"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:07:13.657387 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:13.657334    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a31f57ff-3195-43ba-8158-99b5a3728300-kube-api-access-ngkbq" pod "a31f57ff-3195-43ba-8158-99b5a3728300" (UID: "a31f57ff-3195-43ba-8158-99b5a3728300"). InnerVolumeSpecName "kube-api-access-ngkbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:13.756461 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:13.756424    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31f57ff-3195-43ba-8158-99b5a3728300-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:13.756461 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:13.756462    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngkbq\" (UniqueName: \"kubernetes.io/projected/a31f57ff-3195-43ba-8158-99b5a3728300-kube-api-access-ngkbq\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:13.756693 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:13.756477    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31f57ff-3195-43ba-8158-99b5a3728300-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:14.324768 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:14.324728    5639 patch_prober.go:29] interesting pod/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Nov 02 02:07:14.324966 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:14.324792    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g" podUID="2e47ced9-b31c-4196-8b5c-0e980cf9297e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Nov 02 02:07:14.457229 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:14.457187    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg" event={"ID":"a31f57ff-3195-43ba-8158-99b5a3728300","Type":"ContainerDied","Data":"9fa0b0e8d14f4336584971dcdfe1b60b78f7ee4efc255237df6a57b7f06bf073"}
Nov 02 02:07:14.457486 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:14.457242    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fa0b0e8d14f4336584971dcdfe1b60b78f7ee4efc255237df6a57b7f06bf073"
Nov 02 02:07:14.457486 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:14.457239    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg"
Nov 02 02:07:15.330969 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:15.330938    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:07:15.331448 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:15.331336    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:07:15.332138 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:15.332116    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:07:15.446895 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:15.446857    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:07:17.645311 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:17.645258    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd"]
Nov 02 02:07:21.935720 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:21.935685    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl"]
Nov 02 02:07:22.044419 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:22.044379    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm"]
Nov 02 02:07:22.646066 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:22.646003    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd" podUID="7e5a265a-31f7-4042-88cc-a18783a14ecb" containerName="payload-processing" containerID="cri-o://d406f0b5df890b98c315f9e3a59237bc9bf3213c0ec931117c0dbd959c6162ec" gracePeriod=25
Nov 02 02:07:24.532743 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:24.532699    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd" podUID="7e5a265a-31f7-4042-88cc-a18783a14ecb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:07:25.502517 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:25.502472    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main" containerID="cri-o://227cdb9dc495fe33b66b2dbabe9dccf7cd2c3698beda12c383961a378665f37f" gracePeriod=45
Nov 02 02:07:25.782272 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:25.782246    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 02:07:25.849221 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:25.849191    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-kube-api-access-ps827\" (UniqueName: \"kubernetes.io/projected/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-kube-api-access-ps827\") pod \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\" (UID: \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\") "
Nov 02 02:07:25.849485 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:25.849298    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-dshm\") pod \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\" (UID: \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\") "
Nov 02 02:07:25.849485 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:25.849447    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-home\" (UniqueName: \"kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-home\") pod \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\" (UID: \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\") "
Nov 02 02:07:25.849630 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:25.849506    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-tmp-dir\") pod \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\" (UID: \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\") "
Nov 02 02:07:25.849630 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:25.849575    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-tls-certs\" (UniqueName: \"kubernetes.io/secret/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-tls-certs\") pod \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\" (UID: \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\") "
Nov 02 02:07:25.849738 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:25.849667    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-model-cache\") pod \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\" (UID: \"43bdf2ad-6e64-46da-ad9f-4b9dc775290a\") "
Nov 02 02:07:25.849937 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:25.849830    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-tmp-dir" pod "43bdf2ad-6e64-46da-ad9f-4b9dc775290a" (UID: "43bdf2ad-6e64-46da-ad9f-4b9dc775290a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:25.850213 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:25.850195    5639 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-tmp-dir\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:25.850334 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:25.850261    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-home" pod "43bdf2ad-6e64-46da-ad9f-4b9dc775290a" (UID: "43bdf2ad-6e64-46da-ad9f-4b9dc775290a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:25.850714 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:25.850664    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-model-cache" pod "43bdf2ad-6e64-46da-ad9f-4b9dc775290a" (UID: "43bdf2ad-6e64-46da-ad9f-4b9dc775290a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:25.853083 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:25.853042    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-kube-api-access-ps827" pod "43bdf2ad-6e64-46da-ad9f-4b9dc775290a" (UID: "43bdf2ad-6e64-46da-ad9f-4b9dc775290a"). InnerVolumeSpecName "kube-api-access-ps827". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:25.855491 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:25.855185    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-tls-certs" pod "43bdf2ad-6e64-46da-ad9f-4b9dc775290a" (UID: "43bdf2ad-6e64-46da-ad9f-4b9dc775290a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:07:25.855724 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:25.855699    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-dshm" pod "43bdf2ad-6e64-46da-ad9f-4b9dc775290a" (UID: "43bdf2ad-6e64-46da-ad9f-4b9dc775290a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:25.951391 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:25.951350    5639 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-dshm\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:25.951391 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:25.951380    5639 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-home\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:25.951391 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:25.951389    5639 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-tls-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:25.951391 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:25.951398    5639 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-model-cache\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:25.951778 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:25.951407    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps827\" (UniqueName: \"kubernetes.io/projected/43bdf2ad-6e64-46da-ad9f-4b9dc775290a-kube-api-access-ps827\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:26.565388 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:26.565353    5639 generic.go:356] "Generic (PLEG): container finished" podID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerID="227cdb9dc495fe33b66b2dbabe9dccf7cd2c3698beda12c383961a378665f37f" exitCode=0
Nov 02 02:07:26.565563 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:26.565418    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"
Nov 02 02:07:26.565563 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:26.565442    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" event={"ID":"43bdf2ad-6e64-46da-ad9f-4b9dc775290a","Type":"ContainerDied","Data":"227cdb9dc495fe33b66b2dbabe9dccf7cd2c3698beda12c383961a378665f37f"}
Nov 02 02:07:26.565563 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:26.565489    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr" event={"ID":"43bdf2ad-6e64-46da-ad9f-4b9dc775290a","Type":"ContainerDied","Data":"cd039704ac51723a29a50cf45bd6466a074567b49cea925c594416ef27ddecdf"}
Nov 02 02:07:26.565563 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:26.565507    5639 scope.go:122] "RemoveContainer" containerID="227cdb9dc495fe33b66b2dbabe9dccf7cd2c3698beda12c383961a378665f37f"
Nov 02 02:07:26.576795 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:26.576771    5639 scope.go:122] "RemoveContainer" containerID="227cdb9dc495fe33b66b2dbabe9dccf7cd2c3698beda12c383961a378665f37f"
Nov 02 02:07:26.577071 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:07:26.577050    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"227cdb9dc495fe33b66b2dbabe9dccf7cd2c3698beda12c383961a378665f37f\": container with ID starting with 227cdb9dc495fe33b66b2dbabe9dccf7cd2c3698beda12c383961a378665f37f not found: ID does not exist" containerID="227cdb9dc495fe33b66b2dbabe9dccf7cd2c3698beda12c383961a378665f37f"
Nov 02 02:07:26.577124 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:26.577082    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"227cdb9dc495fe33b66b2dbabe9dccf7cd2c3698beda12c383961a378665f37f"} err="failed to get container status \"227cdb9dc495fe33b66b2dbabe9dccf7cd2c3698beda12c383961a378665f37f\": rpc error: code = NotFound desc = could not find container \"227cdb9dc495fe33b66b2dbabe9dccf7cd2c3698beda12c383961a378665f37f\": container with ID starting with 227cdb9dc495fe33b66b2dbabe9dccf7cd2c3698beda12c383961a378665f37f not found: ID does not exist"
Nov 02 02:07:26.729588 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:26.729552    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"]
Nov 02 02:07:26.779014 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:26.778976    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-36a1e0/simulator-w2-36a1e0-kserve-7b7ffb547f-hrstr"]
Nov 02 02:07:26.937012 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:26.936961    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl" podUID="9bb1c148-f351-4295-892b-03ae51db8912" containerName="payload-processing" containerID="cri-o://a08e953832d201390379266066b625de2908521d1ba35f03fbad46878229dab7" gracePeriod=25
Nov 02 02:07:27.045647 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.045599    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm" podUID="e08af262-d376-4399-8a42-99bd89b2897c" containerName="payload-pre-processing" containerID="cri-o://7697a99f01b6478cbf5d3ca1f03e12ff45e5863f2b479ae303cdff69527f0e35" gracePeriod=25
Nov 02 02:07:27.216798 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.216770    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl"
Nov 02 02:07:27.220470 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.220439    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" path="/var/lib/kubelet/pods/43bdf2ad-6e64-46da-ad9f-4b9dc775290a/volumes"
Nov 02 02:07:27.268101 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.268059    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bb1c148-f351-4295-892b-03ae51db8912-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb1c148-f351-4295-892b-03ae51db8912-plugins-config-volume\") pod \"9bb1c148-f351-4295-892b-03ae51db8912\" (UID: \"9bb1c148-f351-4295-892b-03ae51db8912\") "
Nov 02 02:07:27.268267 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.268203    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb1c148-f351-4295-892b-03ae51db8912-kube-api-access-j59rk\" (UniqueName: \"kubernetes.io/projected/9bb1c148-f351-4295-892b-03ae51db8912-kube-api-access-j59rk\") pod \"9bb1c148-f351-4295-892b-03ae51db8912\" (UID: \"9bb1c148-f351-4295-892b-03ae51db8912\") "
Nov 02 02:07:27.268584 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.268555    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb1c148-f351-4295-892b-03ae51db8912-plugins-config-volume" pod "9bb1c148-f351-4295-892b-03ae51db8912" (UID: "9bb1c148-f351-4295-892b-03ae51db8912"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:27.269408 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.268992    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb1c148-f351-4295-892b-03ae51db8912-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:27.270941 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.270910    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb1c148-f351-4295-892b-03ae51db8912-kube-api-access-j59rk" pod "9bb1c148-f351-4295-892b-03ae51db8912" (UID: "9bb1c148-f351-4295-892b-03ae51db8912"). InnerVolumeSpecName "kube-api-access-j59rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:27.311250 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.311225    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm"
Nov 02 02:07:27.369652 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.369616    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e08af262-d376-4399-8a42-99bd89b2897c-kube-api-access-hpnqs\" (UniqueName: \"kubernetes.io/projected/e08af262-d376-4399-8a42-99bd89b2897c-kube-api-access-hpnqs\") pod \"e08af262-d376-4399-8a42-99bd89b2897c\" (UID: \"e08af262-d376-4399-8a42-99bd89b2897c\") "
Nov 02 02:07:27.369863 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.369818    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e08af262-d376-4399-8a42-99bd89b2897c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e08af262-d376-4399-8a42-99bd89b2897c-plugins-config-volume\") pod \"e08af262-d376-4399-8a42-99bd89b2897c\" (UID: \"e08af262-d376-4399-8a42-99bd89b2897c\") "
Nov 02 02:07:27.370750 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.370334    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e08af262-d376-4399-8a42-99bd89b2897c-plugins-config-volume" pod "e08af262-d376-4399-8a42-99bd89b2897c" (UID: "e08af262-d376-4399-8a42-99bd89b2897c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:27.370750 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.370496    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j59rk\" (UniqueName: \"kubernetes.io/projected/9bb1c148-f351-4295-892b-03ae51db8912-kube-api-access-j59rk\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:27.370750 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.370509    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e08af262-d376-4399-8a42-99bd89b2897c-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:27.372724 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.372693    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e08af262-d376-4399-8a42-99bd89b2897c-kube-api-access-hpnqs" pod "e08af262-d376-4399-8a42-99bd89b2897c" (UID: "e08af262-d376-4399-8a42-99bd89b2897c"). InnerVolumeSpecName "kube-api-access-hpnqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:27.420881 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:07:27.420843    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:07:27.472578 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.472489    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpnqs\" (UniqueName: \"kubernetes.io/projected/e08af262-d376-4399-8a42-99bd89b2897c-kube-api-access-hpnqs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:27.573485 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.573451    5639 generic.go:356] "Generic (PLEG): container finished" podID="e08af262-d376-4399-8a42-99bd89b2897c" containerID="7697a99f01b6478cbf5d3ca1f03e12ff45e5863f2b479ae303cdff69527f0e35" exitCode=0
Nov 02 02:07:27.573684 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.573500    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm"
Nov 02 02:07:27.573684 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.573540    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm" event={"ID":"e08af262-d376-4399-8a42-99bd89b2897c","Type":"ContainerDied","Data":"7697a99f01b6478cbf5d3ca1f03e12ff45e5863f2b479ae303cdff69527f0e35"}
Nov 02 02:07:27.573684 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.573593    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm" event={"ID":"e08af262-d376-4399-8a42-99bd89b2897c","Type":"ContainerDied","Data":"da2bf7b5aa22beff5d271137096e2e4ba7d70ce104f6a704ea2fcc1713a27732"}
Nov 02 02:07:27.573684 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.573622    5639 scope.go:122] "RemoveContainer" containerID="7697a99f01b6478cbf5d3ca1f03e12ff45e5863f2b479ae303cdff69527f0e35"
Nov 02 02:07:27.576480 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.576456    5639 generic.go:356] "Generic (PLEG): container finished" podID="9bb1c148-f351-4295-892b-03ae51db8912" containerID="a08e953832d201390379266066b625de2908521d1ba35f03fbad46878229dab7" exitCode=0
Nov 02 02:07:27.576586 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.576495    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl"
Nov 02 02:07:27.576586 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.576516    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl" event={"ID":"9bb1c148-f351-4295-892b-03ae51db8912","Type":"ContainerDied","Data":"a08e953832d201390379266066b625de2908521d1ba35f03fbad46878229dab7"}
Nov 02 02:07:27.576586 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.576561    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl" event={"ID":"9bb1c148-f351-4295-892b-03ae51db8912","Type":"ContainerDied","Data":"a08fde6f731b935242b1e5ac571b70ed73a90c52926aee5107771274d8e4a498"}
Nov 02 02:07:27.590672 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.590646    5639 scope.go:122] "RemoveContainer" containerID="7697a99f01b6478cbf5d3ca1f03e12ff45e5863f2b479ae303cdff69527f0e35"
Nov 02 02:07:27.591019 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:07:27.590996    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7697a99f01b6478cbf5d3ca1f03e12ff45e5863f2b479ae303cdff69527f0e35\": container with ID starting with 7697a99f01b6478cbf5d3ca1f03e12ff45e5863f2b479ae303cdff69527f0e35 not found: ID does not exist" containerID="7697a99f01b6478cbf5d3ca1f03e12ff45e5863f2b479ae303cdff69527f0e35"
Nov 02 02:07:27.591105 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.591032    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7697a99f01b6478cbf5d3ca1f03e12ff45e5863f2b479ae303cdff69527f0e35"} err="failed to get container status \"7697a99f01b6478cbf5d3ca1f03e12ff45e5863f2b479ae303cdff69527f0e35\": rpc error: code = NotFound desc = could not find container \"7697a99f01b6478cbf5d3ca1f03e12ff45e5863f2b479ae303cdff69527f0e35\": container with ID starting with 7697a99f01b6478cbf5d3ca1f03e12ff45e5863f2b479ae303cdff69527f0e35 not found: ID does not exist"
Nov 02 02:07:27.591105 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.591058    5639 scope.go:122] "RemoveContainer" containerID="a08e953832d201390379266066b625de2908521d1ba35f03fbad46878229dab7"
Nov 02 02:07:27.601224 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.601204    5639 scope.go:122] "RemoveContainer" containerID="a08e953832d201390379266066b625de2908521d1ba35f03fbad46878229dab7"
Nov 02 02:07:27.601531 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:07:27.601509    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a08e953832d201390379266066b625de2908521d1ba35f03fbad46878229dab7\": container with ID starting with a08e953832d201390379266066b625de2908521d1ba35f03fbad46878229dab7 not found: ID does not exist" containerID="a08e953832d201390379266066b625de2908521d1ba35f03fbad46878229dab7"
Nov 02 02:07:27.601601 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.601541    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a08e953832d201390379266066b625de2908521d1ba35f03fbad46878229dab7"} err="failed to get container status \"a08e953832d201390379266066b625de2908521d1ba35f03fbad46878229dab7\": rpc error: code = NotFound desc = could not find container \"a08e953832d201390379266066b625de2908521d1ba35f03fbad46878229dab7\": container with ID starting with a08e953832d201390379266066b625de2908521d1ba35f03fbad46878229dab7 not found: ID does not exist"
Nov 02 02:07:27.739211 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.739129    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl"]
Nov 02 02:07:27.744340 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.744308    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c954d-68d466d9-rrxkl"]
Nov 02 02:07:27.852938 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.852901    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm"]
Nov 02 02:07:27.857887 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:27.857845    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c954d-59ff96cdb7-m4dqm"]
Nov 02 02:07:29.218896 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:29.218864    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bb1c148-f351-4295-892b-03ae51db8912" path="/var/lib/kubelet/pods/9bb1c148-f351-4295-892b-03ae51db8912/volumes"
Nov 02 02:07:29.219367 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:29.219296    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e08af262-d376-4399-8a42-99bd89b2897c" path="/var/lib/kubelet/pods/e08af262-d376-4399-8a42-99bd89b2897c/volumes"
Nov 02 02:07:31.109100 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.109065    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6bebaca-f2l77"]
Nov 02 02:07:31.109762 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.109746    5639 cpu_manager.go:414] "containerMap: removing container" podUID="a31f57ff-3195-43ba-8158-99b5a3728300" containerName="revoke-keys"
Nov 02 02:07:31.109808 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.109767    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a31f57ff-3195-43ba-8158-99b5a3728300" containerName="revoke-keys"
Nov 02 02:07:31.109808 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.109787    5639 cpu_manager.go:414] "containerMap: removing container" podUID="9bb1c148-f351-4295-892b-03ae51db8912" containerName="payload-processing"
Nov 02 02:07:31.109808 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.109792    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb1c148-f351-4295-892b-03ae51db8912" containerName="payload-processing"
Nov 02 02:07:31.109913 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.109820    5639 cpu_manager.go:414] "containerMap: removing container" podUID="e08af262-d376-4399-8a42-99bd89b2897c" containerName="payload-pre-processing"
Nov 02 02:07:31.109913 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.109825    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e08af262-d376-4399-8a42-99bd89b2897c" containerName="payload-pre-processing"
Nov 02 02:07:31.109913 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.109838    5639 cpu_manager.go:414] "containerMap: removing container" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main"
Nov 02 02:07:31.109913 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.109842    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main"
Nov 02 02:07:31.110031 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.109927    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="a31f57ff-3195-43ba-8158-99b5a3728300" containerName="revoke-keys"
Nov 02 02:07:31.110031 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.109938    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb1c148-f351-4295-892b-03ae51db8912" containerName="payload-processing"
Nov 02 02:07:31.110031 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.109947    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="e08af262-d376-4399-8a42-99bd89b2897c" containerName="payload-pre-processing"
Nov 02 02:07:31.110031 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.109955    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="43bdf2ad-6e64-46da-ad9f-4b9dc775290a" containerName="main"
Nov 02 02:07:31.113314 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.113298    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bebaca-f2l77"
Nov 02 02:07:31.165964 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.165927    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63995e36-trjk5"]
Nov 02 02:07:31.168124 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.168069    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:07:31.170526 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.170503    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63995e36-trjk5"
Nov 02 02:07:31.171880 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.171854    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6bebaca-f2l77"]
Nov 02 02:07:31.212270 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.212237    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:07:31.217554 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.217525    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:07:31.219797 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.219767    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/321ead4c-a53e-4ff3-9443-2c076c91b7fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6bebaca-f2l77\" (UID: \"321ead4c-a53e-4ff3-9443-2c076c91b7fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bebaca-f2l77"
Nov 02 02:07:31.219954 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.219841    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwz2l\" (UniqueName: \"kubernetes.io/projected/f606ebbc-224b-457e-a324-6bfcf27c6a1b-kube-api-access-bwz2l\") pod \"maas-api-revoke-sub-63995e36-trjk5\" (UID: \"f606ebbc-224b-457e-a324-6bfcf27c6a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63995e36-trjk5"
Nov 02 02:07:31.219954 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.219931    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89drv\" (UniqueName: \"kubernetes.io/projected/321ead4c-a53e-4ff3-9443-2c076c91b7fc-kube-api-access-89drv\") pod \"maas-api-revoke-sub-e6bebaca-f2l77\" (UID: \"321ead4c-a53e-4ff3-9443-2c076c91b7fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bebaca-f2l77"
Nov 02 02:07:31.220044 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.219965    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f606ebbc-224b-457e-a324-6bfcf27c6a1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63995e36-trjk5\" (UID: \"f606ebbc-224b-457e-a324-6bfcf27c6a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63995e36-trjk5"
Nov 02 02:07:31.220044 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.220024    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f606ebbc-224b-457e-a324-6bfcf27c6a1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63995e36-trjk5\" (UID: \"f606ebbc-224b-457e-a324-6bfcf27c6a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63995e36-trjk5"
Nov 02 02:07:31.220123 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.220078    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/321ead4c-a53e-4ff3-9443-2c076c91b7fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6bebaca-f2l77\" (UID: \"321ead4c-a53e-4ff3-9443-2c076c91b7fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bebaca-f2l77"
Nov 02 02:07:31.227627 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.227600    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63995e36-trjk5"]
Nov 02 02:07:31.268908 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.268877    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:07:31.321089 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.321055    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwz2l\" (UniqueName: \"kubernetes.io/projected/f606ebbc-224b-457e-a324-6bfcf27c6a1b-kube-api-access-bwz2l\") pod \"maas-api-revoke-sub-63995e36-trjk5\" (UID: \"f606ebbc-224b-457e-a324-6bfcf27c6a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63995e36-trjk5"
Nov 02 02:07:31.321253 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.321132    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89drv\" (UniqueName: \"kubernetes.io/projected/321ead4c-a53e-4ff3-9443-2c076c91b7fc-kube-api-access-89drv\") pod \"maas-api-revoke-sub-e6bebaca-f2l77\" (UID: \"321ead4c-a53e-4ff3-9443-2c076c91b7fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bebaca-f2l77"
Nov 02 02:07:31.321253 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.321163    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f606ebbc-224b-457e-a324-6bfcf27c6a1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63995e36-trjk5\" (UID: \"f606ebbc-224b-457e-a324-6bfcf27c6a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63995e36-trjk5"
Nov 02 02:07:31.321253 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.321200    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f606ebbc-224b-457e-a324-6bfcf27c6a1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63995e36-trjk5\" (UID: \"f606ebbc-224b-457e-a324-6bfcf27c6a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63995e36-trjk5"
Nov 02 02:07:31.321486 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.321251    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/321ead4c-a53e-4ff3-9443-2c076c91b7fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6bebaca-f2l77\" (UID: \"321ead4c-a53e-4ff3-9443-2c076c91b7fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bebaca-f2l77"
Nov 02 02:07:31.321486 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.321385    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/321ead4c-a53e-4ff3-9443-2c076c91b7fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6bebaca-f2l77\" (UID: \"321ead4c-a53e-4ff3-9443-2c076c91b7fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bebaca-f2l77"
Nov 02 02:07:31.321980 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.321956    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f606ebbc-224b-457e-a324-6bfcf27c6a1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63995e36-trjk5\" (UID: \"f606ebbc-224b-457e-a324-6bfcf27c6a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63995e36-trjk5"
Nov 02 02:07:31.322078 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.321981    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/321ead4c-a53e-4ff3-9443-2c076c91b7fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6bebaca-f2l77\" (UID: \"321ead4c-a53e-4ff3-9443-2c076c91b7fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bebaca-f2l77"
Nov 02 02:07:31.323902 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.323877    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f606ebbc-224b-457e-a324-6bfcf27c6a1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63995e36-trjk5\" (UID: \"f606ebbc-224b-457e-a324-6bfcf27c6a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63995e36-trjk5"
Nov 02 02:07:31.324005 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.323918    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/321ead4c-a53e-4ff3-9443-2c076c91b7fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6bebaca-f2l77\" (UID: \"321ead4c-a53e-4ff3-9443-2c076c91b7fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bebaca-f2l77"
Nov 02 02:07:31.381973 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.381904    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89drv\" (UniqueName: \"kubernetes.io/projected/321ead4c-a53e-4ff3-9443-2c076c91b7fc-kube-api-access-89drv\") pod \"maas-api-revoke-sub-e6bebaca-f2l77\" (UID: \"321ead4c-a53e-4ff3-9443-2c076c91b7fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bebaca-f2l77"
Nov 02 02:07:31.382108 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.382043    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwz2l\" (UniqueName: \"kubernetes.io/projected/f606ebbc-224b-457e-a324-6bfcf27c6a1b-kube-api-access-bwz2l\") pod \"maas-api-revoke-sub-63995e36-trjk5\" (UID: \"f606ebbc-224b-457e-a324-6bfcf27c6a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63995e36-trjk5"
Nov 02 02:07:31.426484 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.426456    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bebaca-f2l77"
Nov 02 02:07:31.487518 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.487262    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63995e36-trjk5"
Nov 02 02:07:31.807495 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.807455    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6bebaca-f2l77"]
Nov 02 02:07:31.808684 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:07:31.808654    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321ead4c_a53e_4ff3_9443_2c076c91b7fc.slice/crio-a2394e257d95207d067a47751bab51432916b8fdc7e63e42463cc26627edfa7b WatchSource:0}: Error finding container a2394e257d95207d067a47751bab51432916b8fdc7e63e42463cc26627edfa7b: Status 404 returned error can't find the container with id a2394e257d95207d067a47751bab51432916b8fdc7e63e42463cc26627edfa7b
Nov 02 02:07:31.907591 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:31.907564    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63995e36-trjk5"]
Nov 02 02:07:31.909122 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:07:31.909076    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf606ebbc_224b_457e_a324_6bfcf27c6a1b.slice/crio-51407404c98c592ac9db9cf51caa359bc8c9db99bf34099069b43bbeac35afee WatchSource:0}: Error finding container 51407404c98c592ac9db9cf51caa359bc8c9db99bf34099069b43bbeac35afee: Status 404 returned error can't find the container with id 51407404c98c592ac9db9cf51caa359bc8c9db99bf34099069b43bbeac35afee
Nov 02 02:07:32.625266 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:32.625227    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63995e36-trjk5" event={"ID":"f606ebbc-224b-457e-a324-6bfcf27c6a1b","Type":"ContainerStarted","Data":"0f81d2058f0c2535159fe2841192680618c8d48c8a55c1cda272327323733bdb"}
Nov 02 02:07:32.625748 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:32.625320    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63995e36-trjk5" event={"ID":"f606ebbc-224b-457e-a324-6bfcf27c6a1b","Type":"ContainerStarted","Data":"51407404c98c592ac9db9cf51caa359bc8c9db99bf34099069b43bbeac35afee"}
Nov 02 02:07:32.627057 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:32.627028    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bebaca-f2l77" event={"ID":"321ead4c-a53e-4ff3-9443-2c076c91b7fc","Type":"ContainerStarted","Data":"feb27bcb44c3e8b8ec2d3cc84f6eb88cbc40bcaf26ee8b001eb0ccb2f2cbea4e"}
Nov 02 02:07:32.627192 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:32.627071    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bebaca-f2l77" event={"ID":"321ead4c-a53e-4ff3-9443-2c076c91b7fc","Type":"ContainerStarted","Data":"a2394e257d95207d067a47751bab51432916b8fdc7e63e42463cc26627edfa7b"}
Nov 02 02:07:32.742079 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:32.742024    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63995e36-trjk5" podStartSLOduration=1.7420104969999999 podStartE2EDuration="1.742010497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:07:32.740363738 +0000 UTC m=+2517.644045859" watchObservedRunningTime="2026-11-02 02:07:32.742010497 +0000 UTC m=+2517.645692610"
Nov 02 02:07:32.855784 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:32.855724    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bebaca-f2l77" podStartSLOduration=1.855710668 podStartE2EDuration="1.855710668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:07:32.85519166 +0000 UTC m=+2517.758873776" watchObservedRunningTime="2026-11-02 02:07:32.855710668 +0000 UTC m=+2517.759392782"
Nov 02 02:07:33.636201 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:33.636168    5639 generic.go:356] "Generic (PLEG): container finished" podID="f606ebbc-224b-457e-a324-6bfcf27c6a1b" containerID="0f81d2058f0c2535159fe2841192680618c8d48c8a55c1cda272327323733bdb" exitCode=0
Nov 02 02:07:33.636697 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:33.636241    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63995e36-trjk5" event={"ID":"f606ebbc-224b-457e-a324-6bfcf27c6a1b","Type":"ContainerDied","Data":"0f81d2058f0c2535159fe2841192680618c8d48c8a55c1cda272327323733bdb"}
Nov 02 02:07:33.637934 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:33.637909    5639 generic.go:356] "Generic (PLEG): container finished" podID="321ead4c-a53e-4ff3-9443-2c076c91b7fc" containerID="feb27bcb44c3e8b8ec2d3cc84f6eb88cbc40bcaf26ee8b001eb0ccb2f2cbea4e" exitCode=0
Nov 02 02:07:33.638051 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:33.637995    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bebaca-f2l77" event={"ID":"321ead4c-a53e-4ff3-9443-2c076c91b7fc","Type":"ContainerDied","Data":"feb27bcb44c3e8b8ec2d3cc84f6eb88cbc40bcaf26ee8b001eb0ccb2f2cbea4e"}
Nov 02 02:07:34.532672 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:34.532625    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd" podUID="7e5a265a-31f7-4042-88cc-a18783a14ecb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:07:34.823740 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:34.823717    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bebaca-f2l77"
Nov 02 02:07:34.854417 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:34.854388    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63995e36-trjk5"
Nov 02 02:07:34.968843 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:34.968811    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f606ebbc-224b-457e-a324-6bfcf27c6a1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f606ebbc-224b-457e-a324-6bfcf27c6a1b-maas-api-openshift-ca\") pod \"f606ebbc-224b-457e-a324-6bfcf27c6a1b\" (UID: \"f606ebbc-224b-457e-a324-6bfcf27c6a1b\") "
Nov 02 02:07:34.969071 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:34.968866    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f606ebbc-224b-457e-a324-6bfcf27c6a1b-kube-api-access-bwz2l\" (UniqueName: \"kubernetes.io/projected/f606ebbc-224b-457e-a324-6bfcf27c6a1b-kube-api-access-bwz2l\") pod \"f606ebbc-224b-457e-a324-6bfcf27c6a1b\" (UID: \"f606ebbc-224b-457e-a324-6bfcf27c6a1b\") "
Nov 02 02:07:34.969071 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:34.968929    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f606ebbc-224b-457e-a324-6bfcf27c6a1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f606ebbc-224b-457e-a324-6bfcf27c6a1b-maas-api-xks-ca\") pod \"f606ebbc-224b-457e-a324-6bfcf27c6a1b\" (UID: \"f606ebbc-224b-457e-a324-6bfcf27c6a1b\") "
Nov 02 02:07:34.969071 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:34.968957    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/321ead4c-a53e-4ff3-9443-2c076c91b7fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/321ead4c-a53e-4ff3-9443-2c076c91b7fc-maas-api-xks-ca\") pod \"321ead4c-a53e-4ff3-9443-2c076c91b7fc\" (UID: \"321ead4c-a53e-4ff3-9443-2c076c91b7fc\") "
Nov 02 02:07:34.969318 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:34.969072    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/321ead4c-a53e-4ff3-9443-2c076c91b7fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/321ead4c-a53e-4ff3-9443-2c076c91b7fc-maas-api-openshift-ca\") pod \"321ead4c-a53e-4ff3-9443-2c076c91b7fc\" (UID: \"321ead4c-a53e-4ff3-9443-2c076c91b7fc\") "
Nov 02 02:07:34.969318 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:34.969124    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/321ead4c-a53e-4ff3-9443-2c076c91b7fc-kube-api-access-89drv\" (UniqueName: \"kubernetes.io/projected/321ead4c-a53e-4ff3-9443-2c076c91b7fc-kube-api-access-89drv\") pod \"321ead4c-a53e-4ff3-9443-2c076c91b7fc\" (UID: \"321ead4c-a53e-4ff3-9443-2c076c91b7fc\") "
Nov 02 02:07:34.969318 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:34.969266    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f606ebbc-224b-457e-a324-6bfcf27c6a1b-maas-api-openshift-ca" pod "f606ebbc-224b-457e-a324-6bfcf27c6a1b" (UID: "f606ebbc-224b-457e-a324-6bfcf27c6a1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:34.969706 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:34.969652    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/321ead4c-a53e-4ff3-9443-2c076c91b7fc-maas-api-openshift-ca" pod "321ead4c-a53e-4ff3-9443-2c076c91b7fc" (UID: "321ead4c-a53e-4ff3-9443-2c076c91b7fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:34.969931 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:34.969818    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f606ebbc-224b-457e-a324-6bfcf27c6a1b-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:34.969931 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:34.969840    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/321ead4c-a53e-4ff3-9443-2c076c91b7fc-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:34.971651 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:34.971620    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f606ebbc-224b-457e-a324-6bfcf27c6a1b-kube-api-access-bwz2l" pod "f606ebbc-224b-457e-a324-6bfcf27c6a1b" (UID: "f606ebbc-224b-457e-a324-6bfcf27c6a1b"). InnerVolumeSpecName "kube-api-access-bwz2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:34.971776 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:34.971717    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/321ead4c-a53e-4ff3-9443-2c076c91b7fc-maas-api-xks-ca" pod "321ead4c-a53e-4ff3-9443-2c076c91b7fc" (UID: "321ead4c-a53e-4ff3-9443-2c076c91b7fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:07:34.972050 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:34.972027    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321ead4c-a53e-4ff3-9443-2c076c91b7fc-kube-api-access-89drv" pod "321ead4c-a53e-4ff3-9443-2c076c91b7fc" (UID: "321ead4c-a53e-4ff3-9443-2c076c91b7fc"). InnerVolumeSpecName "kube-api-access-89drv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:34.972127 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:34.972106    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f606ebbc-224b-457e-a324-6bfcf27c6a1b-maas-api-xks-ca" pod "f606ebbc-224b-457e-a324-6bfcf27c6a1b" (UID: "f606ebbc-224b-457e-a324-6bfcf27c6a1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:07:35.070972 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:35.070891    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwz2l\" (UniqueName: \"kubernetes.io/projected/f606ebbc-224b-457e-a324-6bfcf27c6a1b-kube-api-access-bwz2l\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:35.070972 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:35.070922    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f606ebbc-224b-457e-a324-6bfcf27c6a1b-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:35.070972 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:35.070931    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/321ead4c-a53e-4ff3-9443-2c076c91b7fc-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:35.070972 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:35.070940    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89drv\" (UniqueName: \"kubernetes.io/projected/321ead4c-a53e-4ff3-9443-2c076c91b7fc-kube-api-access-89drv\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:35.658253 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:35.658218    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63995e36-trjk5"
Nov 02 02:07:35.658491 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:35.658216    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63995e36-trjk5" event={"ID":"f606ebbc-224b-457e-a324-6bfcf27c6a1b","Type":"ContainerDied","Data":"51407404c98c592ac9db9cf51caa359bc8c9db99bf34099069b43bbeac35afee"}
Nov 02 02:07:35.658491 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:35.658431    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51407404c98c592ac9db9cf51caa359bc8c9db99bf34099069b43bbeac35afee"
Nov 02 02:07:35.660024 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:35.660003    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bebaca-f2l77"
Nov 02 02:07:35.660167 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:35.660017    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bebaca-f2l77" event={"ID":"321ead4c-a53e-4ff3-9443-2c076c91b7fc","Type":"ContainerDied","Data":"a2394e257d95207d067a47751bab51432916b8fdc7e63e42463cc26627edfa7b"}
Nov 02 02:07:35.660167 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:35.660064    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2394e257d95207d067a47751bab51432916b8fdc7e63e42463cc26627edfa7b"
Nov 02 02:07:38.527552 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:38.527514    5639 scope.go:122] "RemoveContainer" containerID="bbdc3686ddfbe096ef29e3a4eea470298e991524cbb4e031154daad1706f3ce8"
Nov 02 02:07:38.538792 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:38.538762    5639 scope.go:122] "RemoveContainer" containerID="a55b4895ff4b997821a3ab58d9b81775f037ce3f599ec9aa3dfbfbf400bb0821"
Nov 02 02:07:38.548637 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:38.548618    5639 scope.go:122] "RemoveContainer" containerID="897f3fa51e2f6a6ae30aaad99dca841ffb8917b8ccede2b668f5288709661e3d"
Nov 02 02:07:43.080989 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:43.080584    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd"
Nov 02 02:07:43.241475 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:43.241441    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e5a265a-31f7-4042-88cc-a18783a14ecb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5a265a-31f7-4042-88cc-a18783a14ecb-plugins-config-volume\") pod \"7e5a265a-31f7-4042-88cc-a18783a14ecb\" (UID: \"7e5a265a-31f7-4042-88cc-a18783a14ecb\") "
Nov 02 02:07:43.242004 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:43.241973    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e5a265a-31f7-4042-88cc-a18783a14ecb-kube-api-access-v5qr2\" (UniqueName: \"kubernetes.io/projected/7e5a265a-31f7-4042-88cc-a18783a14ecb-kube-api-access-v5qr2\") pod \"7e5a265a-31f7-4042-88cc-a18783a14ecb\" (UID: \"7e5a265a-31f7-4042-88cc-a18783a14ecb\") "
Nov 02 02:07:43.242556 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:43.242525    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e5a265a-31f7-4042-88cc-a18783a14ecb-plugins-config-volume" pod "7e5a265a-31f7-4042-88cc-a18783a14ecb" (UID: "7e5a265a-31f7-4042-88cc-a18783a14ecb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:43.242791 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:43.242769    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5a265a-31f7-4042-88cc-a18783a14ecb-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:43.247113 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:43.247081    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5a265a-31f7-4042-88cc-a18783a14ecb-kube-api-access-v5qr2" pod "7e5a265a-31f7-4042-88cc-a18783a14ecb" (UID: "7e5a265a-31f7-4042-88cc-a18783a14ecb"). InnerVolumeSpecName "kube-api-access-v5qr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:43.344043 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:43.344011    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5qr2\" (UniqueName: \"kubernetes.io/projected/7e5a265a-31f7-4042-88cc-a18783a14ecb-kube-api-access-v5qr2\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:43.733312 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:43.733236    5639 generic.go:356] "Generic (PLEG): container finished" podID="7e5a265a-31f7-4042-88cc-a18783a14ecb" containerID="d406f0b5df890b98c315f9e3a59237bc9bf3213c0ec931117c0dbd959c6162ec" exitCode=0
Nov 02 02:07:43.733496 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:43.733358    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd"
Nov 02 02:07:43.733496 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:43.733406    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd" event={"ID":"7e5a265a-31f7-4042-88cc-a18783a14ecb","Type":"ContainerDied","Data":"d406f0b5df890b98c315f9e3a59237bc9bf3213c0ec931117c0dbd959c6162ec"}
Nov 02 02:07:43.733496 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:43.733460    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd" event={"ID":"7e5a265a-31f7-4042-88cc-a18783a14ecb","Type":"ContainerDied","Data":"cb86a80b369ee768b100040f424e35577f5bbe6fcff65808c7d670110d4f3fda"}
Nov 02 02:07:43.733496 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:43.733482    5639 scope.go:122] "RemoveContainer" containerID="d406f0b5df890b98c315f9e3a59237bc9bf3213c0ec931117c0dbd959c6162ec"
Nov 02 02:07:43.744001 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:43.743979    5639 scope.go:122] "RemoveContainer" containerID="d406f0b5df890b98c315f9e3a59237bc9bf3213c0ec931117c0dbd959c6162ec"
Nov 02 02:07:43.744335 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:07:43.744306    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d406f0b5df890b98c315f9e3a59237bc9bf3213c0ec931117c0dbd959c6162ec\": container with ID starting with d406f0b5df890b98c315f9e3a59237bc9bf3213c0ec931117c0dbd959c6162ec not found: ID does not exist" containerID="d406f0b5df890b98c315f9e3a59237bc9bf3213c0ec931117c0dbd959c6162ec"
Nov 02 02:07:43.744471 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:43.744346    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d406f0b5df890b98c315f9e3a59237bc9bf3213c0ec931117c0dbd959c6162ec"} err="failed to get container status \"d406f0b5df890b98c315f9e3a59237bc9bf3213c0ec931117c0dbd959c6162ec\": rpc error: code = NotFound desc = could not find container \"d406f0b5df890b98c315f9e3a59237bc9bf3213c0ec931117c0dbd959c6162ec\": container with ID starting with d406f0b5df890b98c315f9e3a59237bc9bf3213c0ec931117c0dbd959c6162ec not found: ID does not exist"
Nov 02 02:07:43.900087 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:43.900050    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd"]
Nov 02 02:07:43.905575 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:43.905546    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-36a1e0-7f849dd67-bkftd"]
Nov 02 02:07:45.221250 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:45.221212    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e5a265a-31f7-4042-88cc-a18783a14ecb" path="/var/lib/kubelet/pods/7e5a265a-31f7-4042-88cc-a18783a14ecb/volumes"
Nov 02 02:07:56.913540 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:56.913504    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-76e530-wcs89"]
Nov 02 02:07:56.914120 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:56.914105    5639 cpu_manager.go:414] "containerMap: removing container" podUID="f606ebbc-224b-457e-a324-6bfcf27c6a1b" containerName="revoke-keys"
Nov 02 02:07:56.914166 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:56.914121    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f606ebbc-224b-457e-a324-6bfcf27c6a1b" containerName="revoke-keys"
Nov 02 02:07:56.914203 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:56.914166    5639 cpu_manager.go:414] "containerMap: removing container" podUID="321ead4c-a53e-4ff3-9443-2c076c91b7fc" containerName="revoke-keys"
Nov 02 02:07:56.914203 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:56.914172    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="321ead4c-a53e-4ff3-9443-2c076c91b7fc" containerName="revoke-keys"
Nov 02 02:07:56.914203 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:56.914185    5639 cpu_manager.go:414] "containerMap: removing container" podUID="7e5a265a-31f7-4042-88cc-a18783a14ecb" containerName="payload-processing"
Nov 02 02:07:56.914203 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:56.914191    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e5a265a-31f7-4042-88cc-a18783a14ecb" containerName="payload-processing"
Nov 02 02:07:56.914344 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:56.914268    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="321ead4c-a53e-4ff3-9443-2c076c91b7fc" containerName="revoke-keys"
Nov 02 02:07:56.914344 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:56.914296    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e5a265a-31f7-4042-88cc-a18783a14ecb" containerName="payload-processing"
Nov 02 02:07:56.914344 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:56.914306    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="f606ebbc-224b-457e-a324-6bfcf27c6a1b" containerName="revoke-keys"
Nov 02 02:07:56.918929 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:56.918909    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-76e530-wcs89"
Nov 02 02:07:57.015884 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:57.015851    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-76e530-wcs89"]
Nov 02 02:07:57.017465 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:57.017439    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mcmn\" (UniqueName: \"kubernetes.io/projected/c8c363fc-bfe1-4102-9250-971837638835-kube-api-access-4mcmn\") pod \"maas-api-revoke-keys-e2e-api-a-76e530-wcs89\" (UID: \"c8c363fc-bfe1-4102-9250-971837638835\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-76e530-wcs89"
Nov 02 02:07:57.017652 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:57.017623    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c363fc-bfe1-4102-9250-971837638835-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-76e530-wcs89\" (UID: \"c8c363fc-bfe1-4102-9250-971837638835\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-76e530-wcs89"
Nov 02 02:07:57.017723 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:57.017704    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c363fc-bfe1-4102-9250-971837638835-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-76e530-wcs89\" (UID: \"c8c363fc-bfe1-4102-9250-971837638835\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-76e530-wcs89"
Nov 02 02:07:57.020339 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:57.020265    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:07:57.020339 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:57.020315    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:07:57.020520 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:57.020396    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:07:57.020520 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:57.020473    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:07:57.119290 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:57.119249    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mcmn\" (UniqueName: \"kubernetes.io/projected/c8c363fc-bfe1-4102-9250-971837638835-kube-api-access-4mcmn\") pod \"maas-api-revoke-keys-e2e-api-a-76e530-wcs89\" (UID: \"c8c363fc-bfe1-4102-9250-971837638835\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-76e530-wcs89"
Nov 02 02:07:57.119524 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:57.119407    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c363fc-bfe1-4102-9250-971837638835-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-76e530-wcs89\" (UID: \"c8c363fc-bfe1-4102-9250-971837638835\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-76e530-wcs89"
Nov 02 02:07:57.119524 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:57.119444    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c363fc-bfe1-4102-9250-971837638835-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-76e530-wcs89\" (UID: \"c8c363fc-bfe1-4102-9250-971837638835\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-76e530-wcs89"
Nov 02 02:07:57.120169 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:57.120143    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c363fc-bfe1-4102-9250-971837638835-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-76e530-wcs89\" (UID: \"c8c363fc-bfe1-4102-9250-971837638835\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-76e530-wcs89"
Nov 02 02:07:57.122003 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:57.121980    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c363fc-bfe1-4102-9250-971837638835-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-76e530-wcs89\" (UID: \"c8c363fc-bfe1-4102-9250-971837638835\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-76e530-wcs89"
Nov 02 02:07:57.180113 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:57.180040    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mcmn\" (UniqueName: \"kubernetes.io/projected/c8c363fc-bfe1-4102-9250-971837638835-kube-api-access-4mcmn\") pod \"maas-api-revoke-keys-e2e-api-a-76e530-wcs89\" (UID: \"c8c363fc-bfe1-4102-9250-971837638835\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-76e530-wcs89"
Nov 02 02:07:57.233623 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:57.233589    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-76e530-wcs89"
Nov 02 02:07:57.423028 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:07:57.422991    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:07:57.610316 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:57.610267    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-76e530-wcs89"]
Nov 02 02:07:57.611714 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:07:57.611684    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c363fc_bfe1_4102_9250_971837638835.slice/crio-125485d158955f784206f53d3f7e6639fe1174379faf3ef90881995e9358bb6a WatchSource:0}: Error finding container 125485d158955f784206f53d3f7e6639fe1174379faf3ef90881995e9358bb6a: Status 404 returned error can't find the container with id 125485d158955f784206f53d3f7e6639fe1174379faf3ef90881995e9358bb6a
Nov 02 02:07:57.838836 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:57.838800    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-76e530-wcs89" event={"ID":"c8c363fc-bfe1-4102-9250-971837638835","Type":"ContainerStarted","Data":"125485d158955f784206f53d3f7e6639fe1174379faf3ef90881995e9358bb6a"}
Nov 02 02:07:58.848185 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:58.848150    5639 generic.go:356] "Generic (PLEG): container finished" podID="c8c363fc-bfe1-4102-9250-971837638835" containerID="a41e8a2c8550d0ef2345a0c2ea47abb0d884247da8eb4f0ded8970dfbe3b37c7" exitCode=0
Nov 02 02:07:58.848633 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:07:58.848219    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-76e530-wcs89" event={"ID":"c8c363fc-bfe1-4102-9250-971837638835","Type":"ContainerDied","Data":"a41e8a2c8550d0ef2345a0c2ea47abb0d884247da8eb4f0ded8970dfbe3b37c7"}
Nov 02 02:08:00.009470 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:00.009444    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-76e530-wcs89"
Nov 02 02:08:00.159192 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:00.159102    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8c363fc-bfe1-4102-9250-971837638835-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c363fc-bfe1-4102-9250-971837638835-maas-api-xks-ca\") pod \"c8c363fc-bfe1-4102-9250-971837638835\" (UID: \"c8c363fc-bfe1-4102-9250-971837638835\") "
Nov 02 02:08:00.159416 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:00.159245    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8c363fc-bfe1-4102-9250-971837638835-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c363fc-bfe1-4102-9250-971837638835-maas-api-openshift-ca\") pod \"c8c363fc-bfe1-4102-9250-971837638835\" (UID: \"c8c363fc-bfe1-4102-9250-971837638835\") "
Nov 02 02:08:00.159416 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:00.159323    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8c363fc-bfe1-4102-9250-971837638835-kube-api-access-4mcmn\" (UniqueName: \"kubernetes.io/projected/c8c363fc-bfe1-4102-9250-971837638835-kube-api-access-4mcmn\") pod \"c8c363fc-bfe1-4102-9250-971837638835\" (UID: \"c8c363fc-bfe1-4102-9250-971837638835\") "
Nov 02 02:08:00.159715 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:00.159689    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c363fc-bfe1-4102-9250-971837638835-maas-api-openshift-ca" pod "c8c363fc-bfe1-4102-9250-971837638835" (UID: "c8c363fc-bfe1-4102-9250-971837638835"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:00.159912 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:00.159894    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c363fc-bfe1-4102-9250-971837638835-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:00.161739 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:00.161714    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8c363fc-bfe1-4102-9250-971837638835-maas-api-xks-ca" pod "c8c363fc-bfe1-4102-9250-971837638835" (UID: "c8c363fc-bfe1-4102-9250-971837638835"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:08:00.162164 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:00.162144    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c363fc-bfe1-4102-9250-971837638835-kube-api-access-4mcmn" pod "c8c363fc-bfe1-4102-9250-971837638835" (UID: "c8c363fc-bfe1-4102-9250-971837638835"). InnerVolumeSpecName "kube-api-access-4mcmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:00.261459 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:00.261429    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mcmn\" (UniqueName: \"kubernetes.io/projected/c8c363fc-bfe1-4102-9250-971837638835-kube-api-access-4mcmn\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:00.261459 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:00.261456    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c363fc-bfe1-4102-9250-971837638835-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:00.863917 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:00.863880    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-76e530-wcs89" event={"ID":"c8c363fc-bfe1-4102-9250-971837638835","Type":"ContainerDied","Data":"125485d158955f784206f53d3f7e6639fe1174379faf3ef90881995e9358bb6a"}
Nov 02 02:08:00.863917 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:00.863941    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="125485d158955f784206f53d3f7e6639fe1174379faf3ef90881995e9358bb6a"
Nov 02 02:08:00.864266 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:00.863947    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-76e530-wcs89"
Nov 02 02:08:07.141074 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:07.141032    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:08:16.464820 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:16.464634    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"]
Nov 02 02:08:16.467377 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:16.465674    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq" podUID="2da5e48a-b3fd-4b3a-97d9-505aea24dc94" containerName="istio-proxy" containerID="cri-o://65698364ceb649fb3159bbb0894b582246cc6b83cad9652e1d5b40cd7790cb41" gracePeriod=30
Nov 02 02:08:16.512966 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:16.512927    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"]
Nov 02 02:08:16.513186 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:16.513161    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm" podUID="8bf946ac-2c79-4b61-8890-703068477cdf" containerName="istio-proxy" containerID="cri-o://9b916d457af7ad6a3530497528fa9fda585ef1052bcf4cffd69adcfcadbeb668" gracePeriod=30
Nov 02 02:08:19.051404 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:19.051370    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9657df10-g8cdq"]
Nov 02 02:08:19.100762 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:19.100719    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9657df10-g8cdq"]
Nov 02 02:08:19.219027 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:19.218991    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1964d9f-bf5f-444f-a781-3284985a726a" path="/var/lib/kubelet/pods/a1964d9f-bf5f-444f-a781-3284985a726a/volumes"
Nov 02 02:08:20.049463 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:20.049426    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-537ee113-kwdbr"]
Nov 02 02:08:20.066771 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:20.066738    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-537ee113-kwdbr"]
Nov 02 02:08:21.219241 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.219201    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="875bffb8-7c4b-44dc-97fe-3be42c9ffe93" path="/var/lib/kubelet/pods/875bffb8-7c4b-44dc-97fe-3be42c9ffe93/volumes"
Nov 02 02:08:21.598861 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.598829    5639 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Nov 02 02:08:21.599030 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.598884    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq" podUID="2da5e48a-b3fd-4b3a-97d9-505aea24dc94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Nov 02 02:08:21.816053 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.816027    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 02:08:21.843011 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.842971    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 02:08:21.873994 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.873959    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-kube-api-access-8kscz\" (UniqueName: \"kubernetes.io/projected/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-kube-api-access-8kscz\") pod \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") "
Nov 02 02:08:21.873994 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.873996    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-token\" (UniqueName: \"kubernetes.io/projected/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-token\") pod \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") "
Nov 02 02:08:21.874206 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.874042    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-podinfo\") pod \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") "
Nov 02 02:08:21.874206 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.874096    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-envoy\") pod \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") "
Nov 02 02:08:21.874206 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.874178    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-workload-socket\") pod \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") "
Nov 02 02:08:21.874404 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.874213    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istiod-ca-cert\") pod \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") "
Nov 02 02:08:21.874404 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.874293    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-data\") pod \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") "
Nov 02 02:08:21.874404 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.874367    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-credential-socket\") pod \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") "
Nov 02 02:08:21.874569 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.874406    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-workload-certs\") pod \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\" (UID: \"2da5e48a-b3fd-4b3a-97d9-505aea24dc94\") "
Nov 02 02:08:21.874569 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.874547    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-workload-socket" pod "2da5e48a-b3fd-4b3a-97d9-505aea24dc94" (UID: "2da5e48a-b3fd-4b3a-97d9-505aea24dc94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:21.874743 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.874697    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-data" pod "2da5e48a-b3fd-4b3a-97d9-505aea24dc94" (UID: "2da5e48a-b3fd-4b3a-97d9-505aea24dc94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:21.874812 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.874767    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-workload-certs" pod "2da5e48a-b3fd-4b3a-97d9-505aea24dc94" (UID: "2da5e48a-b3fd-4b3a-97d9-505aea24dc94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:21.874948 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.874924    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-credential-socket" pod "2da5e48a-b3fd-4b3a-97d9-505aea24dc94" (UID: "2da5e48a-b3fd-4b3a-97d9-505aea24dc94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:21.875128 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.875106    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istiod-ca-cert" pod "2da5e48a-b3fd-4b3a-97d9-505aea24dc94" (UID: "2da5e48a-b3fd-4b3a-97d9-505aea24dc94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:21.875251 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.875231    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:21.875319 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.875259    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:21.875319 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.875312    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:21.875396 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.875329    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:21.875396 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.875343    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:21.876709 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.876686    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-podinfo" pod "2da5e48a-b3fd-4b3a-97d9-505aea24dc94" (UID: "2da5e48a-b3fd-4b3a-97d9-505aea24dc94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:08:21.876801 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.876754    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-kube-api-access-8kscz" pod "2da5e48a-b3fd-4b3a-97d9-505aea24dc94" (UID: "2da5e48a-b3fd-4b3a-97d9-505aea24dc94"). InnerVolumeSpecName "kube-api-access-8kscz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:21.876891 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.876873    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-token" pod "2da5e48a-b3fd-4b3a-97d9-505aea24dc94" (UID: "2da5e48a-b3fd-4b3a-97d9-505aea24dc94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:21.876946 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.876901    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-envoy" pod "2da5e48a-b3fd-4b3a-97d9-505aea24dc94" (UID: "2da5e48a-b3fd-4b3a-97d9-505aea24dc94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:21.975805 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.975762    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-workload-certs\") pod \"8bf946ac-2c79-4b61-8890-703068477cdf\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") "
Nov 02 02:08:21.975805 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.975812    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-workload-socket\") pod \"8bf946ac-2c79-4b61-8890-703068477cdf\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") "
Nov 02 02:08:21.976075 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.975874    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bf946ac-2c79-4b61-8890-703068477cdf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf946ac-2c79-4b61-8890-703068477cdf-istiod-ca-cert\") pod \"8bf946ac-2c79-4b61-8890-703068477cdf\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") "
Nov 02 02:08:21.976075 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.975918    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8bf946ac-2c79-4b61-8890-703068477cdf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf946ac-2c79-4b61-8890-703068477cdf-istio-podinfo\") pod \"8bf946ac-2c79-4b61-8890-703068477cdf\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") "
Nov 02 02:08:21.976075 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.975965    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf946ac-2c79-4b61-8890-703068477cdf-kube-api-access-pmcr6\" (UniqueName: \"kubernetes.io/projected/8bf946ac-2c79-4b61-8890-703068477cdf-kube-api-access-pmcr6\") pod \"8bf946ac-2c79-4b61-8890-703068477cdf\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") "
Nov 02 02:08:21.976238 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.976079    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-workload-certs" pod "8bf946ac-2c79-4b61-8890-703068477cdf" (UID: "8bf946ac-2c79-4b61-8890-703068477cdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:21.976238 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.976130    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-istio-data\") pod \"8bf946ac-2c79-4b61-8890-703068477cdf\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") "
Nov 02 02:08:21.976238 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.976155    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-workload-socket" pod "8bf946ac-2c79-4b61-8890-703068477cdf" (UID: "8bf946ac-2c79-4b61-8890-703068477cdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:21.976238 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.976226    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-credential-socket\") pod \"8bf946ac-2c79-4b61-8890-703068477cdf\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") "
Nov 02 02:08:21.976574 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.976258    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-istio-envoy\") pod \"8bf946ac-2c79-4b61-8890-703068477cdf\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") "
Nov 02 02:08:21.976574 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.976327    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-istio-data" pod "8bf946ac-2c79-4b61-8890-703068477cdf" (UID: "8bf946ac-2c79-4b61-8890-703068477cdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:21.976574 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.976340    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf946ac-2c79-4b61-8890-703068477cdf-istio-token\" (UniqueName: \"kubernetes.io/projected/8bf946ac-2c79-4b61-8890-703068477cdf-istio-token\") pod \"8bf946ac-2c79-4b61-8890-703068477cdf\" (UID: \"8bf946ac-2c79-4b61-8890-703068477cdf\") "
Nov 02 02:08:21.976574 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.976413    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf946ac-2c79-4b61-8890-703068477cdf-istiod-ca-cert" pod "8bf946ac-2c79-4b61-8890-703068477cdf" (UID: "8bf946ac-2c79-4b61-8890-703068477cdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:21.976574 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.976495    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-credential-socket" pod "8bf946ac-2c79-4b61-8890-703068477cdf" (UID: "8bf946ac-2c79-4b61-8890-703068477cdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:21.977112 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.977049    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:21.977112 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.977080    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:21.977112 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.977096    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:21.977112 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.977111    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf946ac-2c79-4b61-8890-703068477cdf-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:21.977112 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.977126    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kscz\" (UniqueName: \"kubernetes.io/projected/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-kube-api-access-8kscz\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:21.977112 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.977159    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:21.977112 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.977175    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:21.977112 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.977189    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2da5e48a-b3fd-4b3a-97d9-505aea24dc94-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:21.977775 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.977203    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:21.978560 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.978537    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf946ac-2c79-4b61-8890-703068477cdf-kube-api-access-pmcr6" pod "8bf946ac-2c79-4b61-8890-703068477cdf" (UID: "8bf946ac-2c79-4b61-8890-703068477cdf"). InnerVolumeSpecName "kube-api-access-pmcr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:21.978781 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.978762    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf946ac-2c79-4b61-8890-703068477cdf-istio-token" pod "8bf946ac-2c79-4b61-8890-703068477cdf" (UID: "8bf946ac-2c79-4b61-8890-703068477cdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:21.978953 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.978927    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-istio-envoy" pod "8bf946ac-2c79-4b61-8890-703068477cdf" (UID: "8bf946ac-2c79-4b61-8890-703068477cdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:21.979072 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:21.978929    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bf946ac-2c79-4b61-8890-703068477cdf-istio-podinfo" pod "8bf946ac-2c79-4b61-8890-703068477cdf" (UID: "8bf946ac-2c79-4b61-8890-703068477cdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:08:22.052648 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.052553    5639 generic.go:356] "Generic (PLEG): container finished" podID="2da5e48a-b3fd-4b3a-97d9-505aea24dc94" containerID="65698364ceb649fb3159bbb0894b582246cc6b83cad9652e1d5b40cd7790cb41" exitCode=0
Nov 02 02:08:22.052648 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.052611    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"
Nov 02 02:08:22.052895 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.052639    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq" event={"ID":"2da5e48a-b3fd-4b3a-97d9-505aea24dc94","Type":"ContainerDied","Data":"65698364ceb649fb3159bbb0894b582246cc6b83cad9652e1d5b40cd7790cb41"}
Nov 02 02:08:22.052895 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.052706    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq" event={"ID":"2da5e48a-b3fd-4b3a-97d9-505aea24dc94","Type":"ContainerDied","Data":"eb179c3ee8425c2ddfd9c45b85c59c57fd349c9660905e15af50afa17074e735"}
Nov 02 02:08:22.052895 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.052735    5639 scope.go:122] "RemoveContainer" containerID="65698364ceb649fb3159bbb0894b582246cc6b83cad9652e1d5b40cd7790cb41"
Nov 02 02:08:22.054692 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.054657    5639 generic.go:356] "Generic (PLEG): container finished" podID="8bf946ac-2c79-4b61-8890-703068477cdf" containerID="9b916d457af7ad6a3530497528fa9fda585ef1052bcf4cffd69adcfcadbeb668" exitCode=0
Nov 02 02:08:22.054824 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.054717    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"
Nov 02 02:08:22.054824 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.054743    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm" event={"ID":"8bf946ac-2c79-4b61-8890-703068477cdf","Type":"ContainerDied","Data":"9b916d457af7ad6a3530497528fa9fda585ef1052bcf4cffd69adcfcadbeb668"}
Nov 02 02:08:22.054824 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.054802    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm" event={"ID":"8bf946ac-2c79-4b61-8890-703068477cdf","Type":"ContainerDied","Data":"b3bdbd4c37cb5f97a7d0256257a4cc640b28d1f27d8c80fed69317ea08011e0f"}
Nov 02 02:08:22.065859 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.065841    5639 scope.go:122] "RemoveContainer" containerID="fd7ec3605ebe009a0d9fb2a852381c7e698fc9de83f5d8bfd0f56ccc28447ef7"
Nov 02 02:08:22.076362 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.076338    5639 scope.go:122] "RemoveContainer" containerID="65698364ceb649fb3159bbb0894b582246cc6b83cad9652e1d5b40cd7790cb41"
Nov 02 02:08:22.076683 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:08:22.076660    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65698364ceb649fb3159bbb0894b582246cc6b83cad9652e1d5b40cd7790cb41\": container with ID starting with 65698364ceb649fb3159bbb0894b582246cc6b83cad9652e1d5b40cd7790cb41 not found: ID does not exist" containerID="65698364ceb649fb3159bbb0894b582246cc6b83cad9652e1d5b40cd7790cb41"
Nov 02 02:08:22.076779 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.076695    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65698364ceb649fb3159bbb0894b582246cc6b83cad9652e1d5b40cd7790cb41"} err="failed to get container status \"65698364ceb649fb3159bbb0894b582246cc6b83cad9652e1d5b40cd7790cb41\": rpc error: code = NotFound desc = could not find container \"65698364ceb649fb3159bbb0894b582246cc6b83cad9652e1d5b40cd7790cb41\": container with ID starting with 65698364ceb649fb3159bbb0894b582246cc6b83cad9652e1d5b40cd7790cb41 not found: ID does not exist"
Nov 02 02:08:22.076779 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.076714    5639 scope.go:122] "RemoveContainer" containerID="fd7ec3605ebe009a0d9fb2a852381c7e698fc9de83f5d8bfd0f56ccc28447ef7"
Nov 02 02:08:22.077000 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:08:22.076977    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd7ec3605ebe009a0d9fb2a852381c7e698fc9de83f5d8bfd0f56ccc28447ef7\": container with ID starting with fd7ec3605ebe009a0d9fb2a852381c7e698fc9de83f5d8bfd0f56ccc28447ef7 not found: ID does not exist" containerID="fd7ec3605ebe009a0d9fb2a852381c7e698fc9de83f5d8bfd0f56ccc28447ef7"
Nov 02 02:08:22.077089 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.077012    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd7ec3605ebe009a0d9fb2a852381c7e698fc9de83f5d8bfd0f56ccc28447ef7"} err="failed to get container status \"fd7ec3605ebe009a0d9fb2a852381c7e698fc9de83f5d8bfd0f56ccc28447ef7\": rpc error: code = NotFound desc = could not find container \"fd7ec3605ebe009a0d9fb2a852381c7e698fc9de83f5d8bfd0f56ccc28447ef7\": container with ID starting with fd7ec3605ebe009a0d9fb2a852381c7e698fc9de83f5d8bfd0f56ccc28447ef7 not found: ID does not exist"
Nov 02 02:08:22.077089 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.077040    5639 scope.go:122] "RemoveContainer" containerID="9b916d457af7ad6a3530497528fa9fda585ef1052bcf4cffd69adcfcadbeb668"
Nov 02 02:08:22.078104 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.078073    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf946ac-2c79-4b61-8890-703068477cdf-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:22.078104 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.078089    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmcr6\" (UniqueName: \"kubernetes.io/projected/8bf946ac-2c79-4b61-8890-703068477cdf-kube-api-access-pmcr6\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:22.078104 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.078101    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf946ac-2c79-4b61-8890-703068477cdf-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:22.078222 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.078110    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf946ac-2c79-4b61-8890-703068477cdf-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:22.086989 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.086969    5639 scope.go:122] "RemoveContainer" containerID="e95cf8fbdc7e541966df1c6e1eeb15cbc21cc51e2caacb56439a668f1d97211d"
Nov 02 02:08:22.099267 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.099239    5639 scope.go:122] "RemoveContainer" containerID="9b916d457af7ad6a3530497528fa9fda585ef1052bcf4cffd69adcfcadbeb668"
Nov 02 02:08:22.100448 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:08:22.100379    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b916d457af7ad6a3530497528fa9fda585ef1052bcf4cffd69adcfcadbeb668\": container with ID starting with 9b916d457af7ad6a3530497528fa9fda585ef1052bcf4cffd69adcfcadbeb668 not found: ID does not exist" containerID="9b916d457af7ad6a3530497528fa9fda585ef1052bcf4cffd69adcfcadbeb668"
Nov 02 02:08:22.100603 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.100460    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b916d457af7ad6a3530497528fa9fda585ef1052bcf4cffd69adcfcadbeb668"} err="failed to get container status \"9b916d457af7ad6a3530497528fa9fda585ef1052bcf4cffd69adcfcadbeb668\": rpc error: code = NotFound desc = could not find container \"9b916d457af7ad6a3530497528fa9fda585ef1052bcf4cffd69adcfcadbeb668\": container with ID starting with 9b916d457af7ad6a3530497528fa9fda585ef1052bcf4cffd69adcfcadbeb668 not found: ID does not exist"
Nov 02 02:08:22.100603 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.100485    5639 scope.go:122] "RemoveContainer" containerID="e95cf8fbdc7e541966df1c6e1eeb15cbc21cc51e2caacb56439a668f1d97211d"
Nov 02 02:08:22.101111 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:08:22.101084    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e95cf8fbdc7e541966df1c6e1eeb15cbc21cc51e2caacb56439a668f1d97211d\": container with ID starting with e95cf8fbdc7e541966df1c6e1eeb15cbc21cc51e2caacb56439a668f1d97211d not found: ID does not exist" containerID="e95cf8fbdc7e541966df1c6e1eeb15cbc21cc51e2caacb56439a668f1d97211d"
Nov 02 02:08:22.101223 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.101120    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e95cf8fbdc7e541966df1c6e1eeb15cbc21cc51e2caacb56439a668f1d97211d"} err="failed to get container status \"e95cf8fbdc7e541966df1c6e1eeb15cbc21cc51e2caacb56439a668f1d97211d\": rpc error: code = NotFound desc = could not find container \"e95cf8fbdc7e541966df1c6e1eeb15cbc21cc51e2caacb56439a668f1d97211d\": container with ID starting with e95cf8fbdc7e541966df1c6e1eeb15cbc21cc51e2caacb56439a668f1d97211d not found: ID does not exist"
Nov 02 02:08:22.231586 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.231548    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"]
Nov 02 02:08:22.236384 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.236342    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-mtbxm"]
Nov 02 02:08:22.343352 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.343319    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"]
Nov 02 02:08:22.348166 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:22.348140    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-l7blq"]
Nov 02 02:08:23.219148 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:23.219115    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2da5e48a-b3fd-4b3a-97d9-505aea24dc94" path="/var/lib/kubelet/pods/2da5e48a-b3fd-4b3a-97d9-505aea24dc94/volumes"
Nov 02 02:08:23.219639 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:23.219624    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bf946ac-2c79-4b61-8890-703068477cdf" path="/var/lib/kubelet/pods/8bf946ac-2c79-4b61-8890-703068477cdf/volumes"
Nov 02 02:08:27.422680 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:08:27.422643    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:08:27.973294 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:27.973249    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf"]
Nov 02 02:08:28.022254 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:28.022219    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-36a1e0-xsvqf"]
Nov 02 02:08:29.057575 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:29.057537    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0933482-xm6kf"]
Nov 02 02:08:29.107812 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:29.107765    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495306e4-5h4xx"]
Nov 02 02:08:29.110472 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:29.110453    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0933482-xm6kf"]
Nov 02 02:08:29.112671 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:29.112649    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495306e4-5h4xx"]
Nov 02 02:08:29.219466 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:29.219415    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32de37c2-a071-4294-97e1-0e17a83db12c" path="/var/lib/kubelet/pods/32de37c2-a071-4294-97e1-0e17a83db12c/volumes"
Nov 02 02:08:29.219991 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:29.219976    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0" path="/var/lib/kubelet/pods/a7c1ca19-5973-405c-9a94-8a1e9d0d8cc0/volumes"
Nov 02 02:08:29.220382 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:29.220368    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1f67b18-24ec-42ea-b266-97bf205fc145" path="/var/lib/kubelet/pods/b1f67b18-24ec-42ea-b266-97bf205fc145/volumes"
Nov 02 02:08:34.144016 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:34.143985    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg"]
Nov 02 02:08:34.197101 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:34.196158    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c954d-xvrqg"]
Nov 02 02:08:35.220218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:35.220188    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a31f57ff-3195-43ba-8158-99b5a3728300" path="/var/lib/kubelet/pods/a31f57ff-3195-43ba-8158-99b5a3728300/volumes"
Nov 02 02:08:38.261756 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:38.261721    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:08:38.598593 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:38.598557    5639 scope.go:122] "RemoveContainer" containerID="4ae07a32499e951453baf1ba687644250b8354ae3787ef94136cbabc2d5666c1"
Nov 02 02:08:38.609485 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:38.609465    5639 scope.go:122] "RemoveContainer" containerID="c855820ba3da811f33f8bff6afd3aaf04d22b6e9980eae77a831031e6b5b63de"
Nov 02 02:08:38.621087 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:38.621059    5639 scope.go:122] "RemoveContainer" containerID="2a0d8daaa97baca2065eaa3a704391d2149e685e823670cf017e1e1796380898"
Nov 02 02:08:38.633016 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:38.632987    5639 scope.go:122] "RemoveContainer" containerID="fa560a3e3809883d470202ce02868185ebf1606d92626d5f39aa82eb781fdbbd"
Nov 02 02:08:38.647794 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:38.647765    5639 scope.go:122] "RemoveContainer" containerID="303e529c7e536e658c21bfab6cbc318624484d964f891c37af7d3f76805d8fe6"
Nov 02 02:08:41.405535 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:41.405503    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"]
Nov 02 02:08:41.405983 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:41.405720    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd" podUID="dae611da-8694-4348-9dff-90bd553e95ec" containerName="istio-proxy" containerID="cri-o://49a2dc519003988e86ea98ed979c9627e2cf982073fe8a6a1e10fc5af6ccc46f" gracePeriod=30
Nov 02 02:08:41.512232 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:41.512195    5639 patch_prober.go:29] interesting pod/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:08:41.512554 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:41.512524    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd" podUID="dae611da-8694-4348-9dff-90bd553e95ec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:08:42.188113 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.188061    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c78625-wch9w"]
Nov 02 02:08:42.189624 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.189596    5639 cpu_manager.go:414] "containerMap: removing container" podUID="2da5e48a-b3fd-4b3a-97d9-505aea24dc94" containerName="istio-proxy"
Nov 02 02:08:42.189624 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.189624    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da5e48a-b3fd-4b3a-97d9-505aea24dc94" containerName="istio-proxy"
Nov 02 02:08:42.189870 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.189714    5639 cpu_manager.go:414] "containerMap: removing container" podUID="c8c363fc-bfe1-4102-9250-971837638835" containerName="revoke-keys"
Nov 02 02:08:42.189870 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.189722    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8c363fc-bfe1-4102-9250-971837638835" containerName="revoke-keys"
Nov 02 02:08:42.189870 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.189747    5639 cpu_manager.go:414] "containerMap: removing container" podUID="8bf946ac-2c79-4b61-8890-703068477cdf" containerName="istio-proxy"
Nov 02 02:08:42.189870 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.189754    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf946ac-2c79-4b61-8890-703068477cdf" containerName="istio-proxy"
Nov 02 02:08:42.189870 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.189780    5639 cpu_manager.go:414] "containerMap: removing container" podUID="8bf946ac-2c79-4b61-8890-703068477cdf" containerName="istio-proxy"
Nov 02 02:08:42.189870 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.189788    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf946ac-2c79-4b61-8890-703068477cdf" containerName="istio-proxy"
Nov 02 02:08:42.190160 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.190091    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da5e48a-b3fd-4b3a-97d9-505aea24dc94" containerName="istio-proxy"
Nov 02 02:08:42.190160 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.190117    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8c363fc-bfe1-4102-9250-971837638835" containerName="revoke-keys"
Nov 02 02:08:42.190160 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.190136    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf946ac-2c79-4b61-8890-703068477cdf" containerName="istio-proxy"
Nov 02 02:08:42.190160 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.190154    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf946ac-2c79-4b61-8890-703068477cdf" containerName="istio-proxy"
Nov 02 02:08:42.198205 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.198180    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c78625-wch9w"
Nov 02 02:08:42.248966 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.248923    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c78625-wch9w"]
Nov 02 02:08:42.250711 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.250688    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:08:42.250711 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.250698    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:08:42.250908 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.250713    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:08:42.250908 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.250746    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:08:42.325175 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.325138    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a777f857-bab6-465f-b718-d3d9b2d22716-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c78625-wch9w\" (UID: \"a777f857-bab6-465f-b718-d3d9b2d22716\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c78625-wch9w"
Nov 02 02:08:42.325175 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.325180    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb7pf\" (UniqueName: \"kubernetes.io/projected/a777f857-bab6-465f-b718-d3d9b2d22716-kube-api-access-cb7pf\") pod \"maas-api-revoke-keys-e2e-api-b-c78625-wch9w\" (UID: \"a777f857-bab6-465f-b718-d3d9b2d22716\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c78625-wch9w"
Nov 02 02:08:42.325451 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.325323    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a777f857-bab6-465f-b718-d3d9b2d22716-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c78625-wch9w\" (UID: \"a777f857-bab6-465f-b718-d3d9b2d22716\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c78625-wch9w"
Nov 02 02:08:42.427052 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.427012    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a777f857-bab6-465f-b718-d3d9b2d22716-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c78625-wch9w\" (UID: \"a777f857-bab6-465f-b718-d3d9b2d22716\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c78625-wch9w"
Nov 02 02:08:42.427495 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.427172    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a777f857-bab6-465f-b718-d3d9b2d22716-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c78625-wch9w\" (UID: \"a777f857-bab6-465f-b718-d3d9b2d22716\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c78625-wch9w"
Nov 02 02:08:42.427495 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.427196    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb7pf\" (UniqueName: \"kubernetes.io/projected/a777f857-bab6-465f-b718-d3d9b2d22716-kube-api-access-cb7pf\") pod \"maas-api-revoke-keys-e2e-api-b-c78625-wch9w\" (UID: \"a777f857-bab6-465f-b718-d3d9b2d22716\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c78625-wch9w"
Nov 02 02:08:42.427827 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.427806    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a777f857-bab6-465f-b718-d3d9b2d22716-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c78625-wch9w\" (UID: \"a777f857-bab6-465f-b718-d3d9b2d22716\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c78625-wch9w"
Nov 02 02:08:42.429631 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.429610    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a777f857-bab6-465f-b718-d3d9b2d22716-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c78625-wch9w\" (UID: \"a777f857-bab6-465f-b718-d3d9b2d22716\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c78625-wch9w"
Nov 02 02:08:42.488128 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.488057    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb7pf\" (UniqueName: \"kubernetes.io/projected/a777f857-bab6-465f-b718-d3d9b2d22716-kube-api-access-cb7pf\") pod \"maas-api-revoke-keys-e2e-api-b-c78625-wch9w\" (UID: \"a777f857-bab6-465f-b718-d3d9b2d22716\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c78625-wch9w"
Nov 02 02:08:42.513418 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.513391    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c78625-wch9w"
Nov 02 02:08:42.895234 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:42.895206    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c78625-wch9w"]
Nov 02 02:08:42.896746 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:08:42.896716    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda777f857_bab6_465f_b718_d3d9b2d22716.slice/crio-75917af61e7dff7de721535ce19c685aa185e25f0ff3b87f24c7873fee369087 WatchSource:0}: Error finding container 75917af61e7dff7de721535ce19c685aa185e25f0ff3b87f24c7873fee369087: Status 404 returned error can't find the container with id 75917af61e7dff7de721535ce19c685aa185e25f0ff3b87f24c7873fee369087
Nov 02 02:08:43.277057 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:43.277018    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c78625-wch9w" event={"ID":"a777f857-bab6-465f-b718-d3d9b2d22716","Type":"ContainerStarted","Data":"d0bbf64018dee4c5ce9f51e50bbc2ab168b86c5d42662568f7f5a8ff2302fcdc"}
Nov 02 02:08:43.277248 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:43.277082    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c78625-wch9w" event={"ID":"a777f857-bab6-465f-b718-d3d9b2d22716","Type":"ContainerStarted","Data":"75917af61e7dff7de721535ce19c685aa185e25f0ff3b87f24c7873fee369087"}
Nov 02 02:08:43.394135 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:43.394081    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c78625-wch9w" podStartSLOduration=1.394067773 podStartE2EDuration="1.394067773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:08:43.392576693 +0000 UTC m=+2588.296258808" watchObservedRunningTime="2026-11-02 02:08:43.394067773 +0000 UTC m=+2588.297749881"
Nov 02 02:08:44.286602 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:44.286564    5639 generic.go:356] "Generic (PLEG): container finished" podID="a777f857-bab6-465f-b718-d3d9b2d22716" containerID="d0bbf64018dee4c5ce9f51e50bbc2ab168b86c5d42662568f7f5a8ff2302fcdc" exitCode=0
Nov 02 02:08:44.287134 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:44.286653    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c78625-wch9w" event={"ID":"a777f857-bab6-465f-b718-d3d9b2d22716","Type":"ContainerDied","Data":"d0bbf64018dee4c5ce9f51e50bbc2ab168b86c5d42662568f7f5a8ff2302fcdc"}
Nov 02 02:08:45.448804 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:45.448779    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c78625-wch9w"
Nov 02 02:08:45.566483 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:45.566361    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a777f857-bab6-465f-b718-d3d9b2d22716-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a777f857-bab6-465f-b718-d3d9b2d22716-maas-api-openshift-ca\") pod \"a777f857-bab6-465f-b718-d3d9b2d22716\" (UID: \"a777f857-bab6-465f-b718-d3d9b2d22716\") "
Nov 02 02:08:45.566483 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:45.566466    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a777f857-bab6-465f-b718-d3d9b2d22716-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a777f857-bab6-465f-b718-d3d9b2d22716-maas-api-xks-ca\") pod \"a777f857-bab6-465f-b718-d3d9b2d22716\" (UID: \"a777f857-bab6-465f-b718-d3d9b2d22716\") "
Nov 02 02:08:45.567158 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:45.566528    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a777f857-bab6-465f-b718-d3d9b2d22716-kube-api-access-cb7pf\" (UniqueName: \"kubernetes.io/projected/a777f857-bab6-465f-b718-d3d9b2d22716-kube-api-access-cb7pf\") pod \"a777f857-bab6-465f-b718-d3d9b2d22716\" (UID: \"a777f857-bab6-465f-b718-d3d9b2d22716\") "
Nov 02 02:08:45.567335 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:45.567304    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a777f857-bab6-465f-b718-d3d9b2d22716-maas-api-openshift-ca" pod "a777f857-bab6-465f-b718-d3d9b2d22716" (UID: "a777f857-bab6-465f-b718-d3d9b2d22716"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:45.569458 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:45.569430    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a777f857-bab6-465f-b718-d3d9b2d22716-maas-api-xks-ca" pod "a777f857-bab6-465f-b718-d3d9b2d22716" (UID: "a777f857-bab6-465f-b718-d3d9b2d22716"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:08:45.570341 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:45.570323    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a777f857-bab6-465f-b718-d3d9b2d22716-kube-api-access-cb7pf" pod "a777f857-bab6-465f-b718-d3d9b2d22716" (UID: "a777f857-bab6-465f-b718-d3d9b2d22716"). InnerVolumeSpecName "kube-api-access-cb7pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:45.668051 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:45.668017    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a777f857-bab6-465f-b718-d3d9b2d22716-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:45.668051 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:45.668051    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a777f857-bab6-465f-b718-d3d9b2d22716-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:45.668051 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:45.668060    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb7pf\" (UniqueName: \"kubernetes.io/projected/a777f857-bab6-465f-b718-d3d9b2d22716-kube-api-access-cb7pf\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:46.301340 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.301312    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c78625-wch9w"
Nov 02 02:08:46.301340 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.301330    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c78625-wch9w" event={"ID":"a777f857-bab6-465f-b718-d3d9b2d22716","Type":"ContainerDied","Data":"75917af61e7dff7de721535ce19c685aa185e25f0ff3b87f24c7873fee369087"}
Nov 02 02:08:46.301565 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.301379    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75917af61e7dff7de721535ce19c685aa185e25f0ff3b87f24c7873fee369087"
Nov 02 02:08:46.676869 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.676829    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:08:46.784661 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.784625    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae611da-8694-4348-9dff-90bd553e95ec-kube-api-access-4qcpn\" (UniqueName: \"kubernetes.io/projected/dae611da-8694-4348-9dff-90bd553e95ec-kube-api-access-4qcpn\") pod \"dae611da-8694-4348-9dff-90bd553e95ec\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") "
Nov 02 02:08:46.784844 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.784709    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-istio-data\") pod \"dae611da-8694-4348-9dff-90bd553e95ec\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") "
Nov 02 02:08:46.784844 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.784748    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dae611da-8694-4348-9dff-90bd553e95ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae611da-8694-4348-9dff-90bd553e95ec-istio-podinfo\") pod \"dae611da-8694-4348-9dff-90bd553e95ec\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") "
Nov 02 02:08:46.784844 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.784808    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-credential-socket\") pod \"dae611da-8694-4348-9dff-90bd553e95ec\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") "
Nov 02 02:08:46.785125 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.784849    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-istio-envoy\") pod \"dae611da-8694-4348-9dff-90bd553e95ec\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") "
Nov 02 02:08:46.785125 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.784944    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae611da-8694-4348-9dff-90bd553e95ec-istio-token\" (UniqueName: \"kubernetes.io/projected/dae611da-8694-4348-9dff-90bd553e95ec-istio-token\") pod \"dae611da-8694-4348-9dff-90bd553e95ec\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") "
Nov 02 02:08:46.785125 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.784988    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dae611da-8694-4348-9dff-90bd553e95ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae611da-8694-4348-9dff-90bd553e95ec-istiod-ca-cert\") pod \"dae611da-8694-4348-9dff-90bd553e95ec\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") "
Nov 02 02:08:46.785125 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.784998    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-istio-data" pod "dae611da-8694-4348-9dff-90bd553e95ec" (UID: "dae611da-8694-4348-9dff-90bd553e95ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:46.785125 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.785027    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-workload-certs\") pod \"dae611da-8694-4348-9dff-90bd553e95ec\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") "
Nov 02 02:08:46.785125 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.785118    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-workload-socket\") pod \"dae611da-8694-4348-9dff-90bd553e95ec\" (UID: \"dae611da-8694-4348-9dff-90bd553e95ec\") "
Nov 02 02:08:46.785471 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.785113    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-credential-socket" pod "dae611da-8694-4348-9dff-90bd553e95ec" (UID: "dae611da-8694-4348-9dff-90bd553e95ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:46.785471 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.785342    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-workload-certs" pod "dae611da-8694-4348-9dff-90bd553e95ec" (UID: "dae611da-8694-4348-9dff-90bd553e95ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:46.785471 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.785399    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-workload-socket" pod "dae611da-8694-4348-9dff-90bd553e95ec" (UID: "dae611da-8694-4348-9dff-90bd553e95ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:46.785626 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.785382    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dae611da-8694-4348-9dff-90bd553e95ec-istiod-ca-cert" pod "dae611da-8694-4348-9dff-90bd553e95ec" (UID: "dae611da-8694-4348-9dff-90bd553e95ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:46.785710 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.785692    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:46.785770 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.785716    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:46.785770 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.785732    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae611da-8694-4348-9dff-90bd553e95ec-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:46.785770 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.785746    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:46.785770 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.785760    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:46.787163 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.787141    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae611da-8694-4348-9dff-90bd553e95ec-kube-api-access-4qcpn" pod "dae611da-8694-4348-9dff-90bd553e95ec" (UID: "dae611da-8694-4348-9dff-90bd553e95ec"). InnerVolumeSpecName "kube-api-access-4qcpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:46.787620 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.787601    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-istio-envoy" pod "dae611da-8694-4348-9dff-90bd553e95ec" (UID: "dae611da-8694-4348-9dff-90bd553e95ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:46.787719 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.787615    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dae611da-8694-4348-9dff-90bd553e95ec-istio-podinfo" pod "dae611da-8694-4348-9dff-90bd553e95ec" (UID: "dae611da-8694-4348-9dff-90bd553e95ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:08:46.787817 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.787798    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae611da-8694-4348-9dff-90bd553e95ec-istio-token" pod "dae611da-8694-4348-9dff-90bd553e95ec" (UID: "dae611da-8694-4348-9dff-90bd553e95ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:46.886688 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.886603    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae611da-8694-4348-9dff-90bd553e95ec-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:46.886688 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.886635    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae611da-8694-4348-9dff-90bd553e95ec-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:46.886688 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.886648    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qcpn\" (UniqueName: \"kubernetes.io/projected/dae611da-8694-4348-9dff-90bd553e95ec-kube-api-access-4qcpn\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:46.886688 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:46.886661    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae611da-8694-4348-9dff-90bd553e95ec-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:47.309645 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:47.309610    5639 generic.go:356] "Generic (PLEG): container finished" podID="dae611da-8694-4348-9dff-90bd553e95ec" containerID="49a2dc519003988e86ea98ed979c9627e2cf982073fe8a6a1e10fc5af6ccc46f" exitCode=0
Nov 02 02:08:47.309817 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:47.309654    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd" event={"ID":"dae611da-8694-4348-9dff-90bd553e95ec","Type":"ContainerDied","Data":"49a2dc519003988e86ea98ed979c9627e2cf982073fe8a6a1e10fc5af6ccc46f"}
Nov 02 02:08:47.309817 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:47.309663    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"
Nov 02 02:08:47.309817 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:47.309700    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd" event={"ID":"dae611da-8694-4348-9dff-90bd553e95ec","Type":"ContainerDied","Data":"1d5fe982ca290531d79bb367087d96c8dbddbab48b2dafd4e6720f55f83ff38b"}
Nov 02 02:08:47.309817 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:47.309725    5639 scope.go:122] "RemoveContainer" containerID="49a2dc519003988e86ea98ed979c9627e2cf982073fe8a6a1e10fc5af6ccc46f"
Nov 02 02:08:47.320204 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:47.320176    5639 scope.go:122] "RemoveContainer" containerID="a942d64662474c14cd716d120f3985690aec6ec7b5411be3468afc15aa06877e"
Nov 02 02:08:47.330573 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:47.330554    5639 scope.go:122] "RemoveContainer" containerID="49a2dc519003988e86ea98ed979c9627e2cf982073fe8a6a1e10fc5af6ccc46f"
Nov 02 02:08:47.330853 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:08:47.330832    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49a2dc519003988e86ea98ed979c9627e2cf982073fe8a6a1e10fc5af6ccc46f\": container with ID starting with 49a2dc519003988e86ea98ed979c9627e2cf982073fe8a6a1e10fc5af6ccc46f not found: ID does not exist" containerID="49a2dc519003988e86ea98ed979c9627e2cf982073fe8a6a1e10fc5af6ccc46f"
Nov 02 02:08:47.330937 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:47.330869    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49a2dc519003988e86ea98ed979c9627e2cf982073fe8a6a1e10fc5af6ccc46f"} err="failed to get container status \"49a2dc519003988e86ea98ed979c9627e2cf982073fe8a6a1e10fc5af6ccc46f\": rpc error: code = NotFound desc = could not find container \"49a2dc519003988e86ea98ed979c9627e2cf982073fe8a6a1e10fc5af6ccc46f\": container with ID starting with 49a2dc519003988e86ea98ed979c9627e2cf982073fe8a6a1e10fc5af6ccc46f not found: ID does not exist"
Nov 02 02:08:47.330937 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:47.330894    5639 scope.go:122] "RemoveContainer" containerID="a942d64662474c14cd716d120f3985690aec6ec7b5411be3468afc15aa06877e"
Nov 02 02:08:47.331163 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:08:47.331145    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a942d64662474c14cd716d120f3985690aec6ec7b5411be3468afc15aa06877e\": container with ID starting with a942d64662474c14cd716d120f3985690aec6ec7b5411be3468afc15aa06877e not found: ID does not exist" containerID="a942d64662474c14cd716d120f3985690aec6ec7b5411be3468afc15aa06877e"
Nov 02 02:08:47.331217 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:47.331170    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a942d64662474c14cd716d120f3985690aec6ec7b5411be3468afc15aa06877e"} err="failed to get container status \"a942d64662474c14cd716d120f3985690aec6ec7b5411be3468afc15aa06877e\": rpc error: code = NotFound desc = could not find container \"a942d64662474c14cd716d120f3985690aec6ec7b5411be3468afc15aa06877e\": container with ID starting with a942d64662474c14cd716d120f3985690aec6ec7b5411be3468afc15aa06877e not found: ID does not exist"
Nov 02 02:08:47.476101 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:47.476067    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"]
Nov 02 02:08:47.525814 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:47.525737    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0c954d-openshift-default-5954667d99-66wvd"]
Nov 02 02:08:49.218803 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:49.218773    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dae611da-8694-4348-9dff-90bd553e95ec" path="/var/lib/kubelet/pods/dae611da-8694-4348-9dff-90bd553e95ec/volumes"
Nov 02 02:08:53.679074 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:53.679040    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd"]
Nov 02 02:08:57.426753 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:08:57.426713    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:08:58.679681 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:58.679629    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd" podUID="4b5f60d5-7e21-48f5-96fb-ad1c519bfae0" containerName="payload-processing" containerID="cri-o://50ebd5d05282e38bdcc5dc6efa7482f651f83226e7fb60836e96bdfa31929482" gracePeriod=25
Nov 02 02:08:58.954701 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:58.954674    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd"
Nov 02 02:08:59.032392 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:59.032356    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b5f60d5-7e21-48f5-96fb-ad1c519bfae0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b5f60d5-7e21-48f5-96fb-ad1c519bfae0-plugins-config-volume\") pod \"4b5f60d5-7e21-48f5-96fb-ad1c519bfae0\" (UID: \"4b5f60d5-7e21-48f5-96fb-ad1c519bfae0\") "
Nov 02 02:08:59.032622 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:59.032501    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b5f60d5-7e21-48f5-96fb-ad1c519bfae0-kube-api-access-qdddh\" (UniqueName: \"kubernetes.io/projected/4b5f60d5-7e21-48f5-96fb-ad1c519bfae0-kube-api-access-qdddh\") pod \"4b5f60d5-7e21-48f5-96fb-ad1c519bfae0\" (UID: \"4b5f60d5-7e21-48f5-96fb-ad1c519bfae0\") "
Nov 02 02:08:59.032778 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:59.032753    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b5f60d5-7e21-48f5-96fb-ad1c519bfae0-plugins-config-volume" pod "4b5f60d5-7e21-48f5-96fb-ad1c519bfae0" (UID: "4b5f60d5-7e21-48f5-96fb-ad1c519bfae0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:59.033027 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:59.033006    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b5f60d5-7e21-48f5-96fb-ad1c519bfae0-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:59.034812 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:59.034790    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b5f60d5-7e21-48f5-96fb-ad1c519bfae0-kube-api-access-qdddh" pod "4b5f60d5-7e21-48f5-96fb-ad1c519bfae0" (UID: "4b5f60d5-7e21-48f5-96fb-ad1c519bfae0"). InnerVolumeSpecName "kube-api-access-qdddh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:59.134421 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:59.134382    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdddh\" (UniqueName: \"kubernetes.io/projected/4b5f60d5-7e21-48f5-96fb-ad1c519bfae0-kube-api-access-qdddh\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:59.406474 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:59.406435    5639 generic.go:356] "Generic (PLEG): container finished" podID="4b5f60d5-7e21-48f5-96fb-ad1c519bfae0" containerID="50ebd5d05282e38bdcc5dc6efa7482f651f83226e7fb60836e96bdfa31929482" exitCode=0
Nov 02 02:08:59.406688 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:59.406485    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd"
Nov 02 02:08:59.406688 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:59.406532    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd" event={"ID":"4b5f60d5-7e21-48f5-96fb-ad1c519bfae0","Type":"ContainerDied","Data":"50ebd5d05282e38bdcc5dc6efa7482f651f83226e7fb60836e96bdfa31929482"}
Nov 02 02:08:59.406688 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:59.406591    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd" event={"ID":"4b5f60d5-7e21-48f5-96fb-ad1c519bfae0","Type":"ContainerDied","Data":"ee0cceb017fa48b5213d09ffe4bc195e04a33ee1b24e2f61716bd57c2737cc62"}
Nov 02 02:08:59.406688 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:59.406623    5639 scope.go:122] "RemoveContainer" containerID="50ebd5d05282e38bdcc5dc6efa7482f651f83226e7fb60836e96bdfa31929482"
Nov 02 02:08:59.417082 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:59.417062    5639 scope.go:122] "RemoveContainer" containerID="50ebd5d05282e38bdcc5dc6efa7482f651f83226e7fb60836e96bdfa31929482"
Nov 02 02:08:59.417449 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:08:59.417424    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50ebd5d05282e38bdcc5dc6efa7482f651f83226e7fb60836e96bdfa31929482\": container with ID starting with 50ebd5d05282e38bdcc5dc6efa7482f651f83226e7fb60836e96bdfa31929482 not found: ID does not exist" containerID="50ebd5d05282e38bdcc5dc6efa7482f651f83226e7fb60836e96bdfa31929482"
Nov 02 02:08:59.417511 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:59.417461    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50ebd5d05282e38bdcc5dc6efa7482f651f83226e7fb60836e96bdfa31929482"} err="failed to get container status \"50ebd5d05282e38bdcc5dc6efa7482f651f83226e7fb60836e96bdfa31929482\": rpc error: code = NotFound desc = could not find container \"50ebd5d05282e38bdcc5dc6efa7482f651f83226e7fb60836e96bdfa31929482\": container with ID starting with 50ebd5d05282e38bdcc5dc6efa7482f651f83226e7fb60836e96bdfa31929482 not found: ID does not exist"
Nov 02 02:08:59.572682 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:59.572593    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd"]
Nov 02 02:08:59.576585 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:08:59.576552    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c78625-7b948bcb6c-dcqxd"]
Nov 02 02:09:01.220191 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:01.220141    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b5f60d5-7e21-48f5-96fb-ad1c519bfae0" path="/var/lib/kubelet/pods/4b5f60d5-7e21-48f5-96fb-ad1c519bfae0/volumes"
Nov 02 02:09:03.929570 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:03.929532    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"]
Nov 02 02:09:03.930454 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:03.930395    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f" podUID="c4443617-f4c0-40c4-a3bf-0b6241d6ce97" containerName="istio-proxy" containerID="cri-o://4cc52e35c4e0543b16494f86403df9c229aeea264497edd5c8cf3b296e658725" gracePeriod=30
Nov 02 02:09:09.203147 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.203074    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 02:09:09.285714 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.285680    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-envoy\") pod \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") "
Nov 02 02:09:09.285945 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.285738    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-kube-api-access-2qb72\" (UniqueName: \"kubernetes.io/projected/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-kube-api-access-2qb72\") pod \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") "
Nov 02 02:09:09.285945 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.285786    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-data\") pod \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") "
Nov 02 02:09:09.285945 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.285836    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-podinfo\") pod \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") "
Nov 02 02:09:09.285945 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.285865    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istiod-ca-cert\") pod \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") "
Nov 02 02:09:09.285945 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.285922    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-workload-socket\") pod \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") "
Nov 02 02:09:09.286305 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.285990    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-workload-certs\") pod \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") "
Nov 02 02:09:09.286305 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.286048    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-credential-socket\") pod \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") "
Nov 02 02:09:09.286305 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.286127    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-data" pod "c4443617-f4c0-40c4-a3bf-0b6241d6ce97" (UID: "c4443617-f4c0-40c4-a3bf-0b6241d6ce97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:09.286501 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.286322    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-workload-certs" pod "c4443617-f4c0-40c4-a3bf-0b6241d6ce97" (UID: "c4443617-f4c0-40c4-a3bf-0b6241d6ce97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:09.286501 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.286129    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-token\" (UniqueName: \"kubernetes.io/projected/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-token\") pod \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\" (UID: \"c4443617-f4c0-40c4-a3bf-0b6241d6ce97\") "
Nov 02 02:09:09.286501 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.286398    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-credential-socket" pod "c4443617-f4c0-40c4-a3bf-0b6241d6ce97" (UID: "c4443617-f4c0-40c4-a3bf-0b6241d6ce97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:09.286501 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.286395    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-workload-socket" pod "c4443617-f4c0-40c4-a3bf-0b6241d6ce97" (UID: "c4443617-f4c0-40c4-a3bf-0b6241d6ce97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:09.286501 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.286433    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istiod-ca-cert" pod "c4443617-f4c0-40c4-a3bf-0b6241d6ce97" (UID: "c4443617-f4c0-40c4-a3bf-0b6241d6ce97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:09:09.287171 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.287147    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:09.287353 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.287175    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:09.287353 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.287189    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:09.287353 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.287202    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:09.287353 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.287216    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:09.288437 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.288414    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-envoy" pod "c4443617-f4c0-40c4-a3bf-0b6241d6ce97" (UID: "c4443617-f4c0-40c4-a3bf-0b6241d6ce97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:09.288797 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.288778    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-token" pod "c4443617-f4c0-40c4-a3bf-0b6241d6ce97" (UID: "c4443617-f4c0-40c4-a3bf-0b6241d6ce97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:09.288938 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.288917    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-podinfo" pod "c4443617-f4c0-40c4-a3bf-0b6241d6ce97" (UID: "c4443617-f4c0-40c4-a3bf-0b6241d6ce97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:09:09.288938 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.288924    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-kube-api-access-2qb72" pod "c4443617-f4c0-40c4-a3bf-0b6241d6ce97" (UID: "c4443617-f4c0-40c4-a3bf-0b6241d6ce97"). InnerVolumeSpecName "kube-api-access-2qb72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:09.389029 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.388987    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:09.389029 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.389030    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qb72\" (UniqueName: \"kubernetes.io/projected/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-kube-api-access-2qb72\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:09.389254 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.389042    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:09.389254 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.389051    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4443617-f4c0-40c4-a3bf-0b6241d6ce97-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:09.497515 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.496540    5639 generic.go:356] "Generic (PLEG): container finished" podID="c4443617-f4c0-40c4-a3bf-0b6241d6ce97" containerID="4cc52e35c4e0543b16494f86403df9c229aeea264497edd5c8cf3b296e658725" exitCode=0
Nov 02 02:09:09.497515 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.496602    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"
Nov 02 02:09:09.497515 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.496605    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f" event={"ID":"c4443617-f4c0-40c4-a3bf-0b6241d6ce97","Type":"ContainerDied","Data":"4cc52e35c4e0543b16494f86403df9c229aeea264497edd5c8cf3b296e658725"}
Nov 02 02:09:09.497515 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.496655    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f" event={"ID":"c4443617-f4c0-40c4-a3bf-0b6241d6ce97","Type":"ContainerDied","Data":"8ce894c17459853e2c06ec6f05e9dc967a0215e1cd84c6050c240a6d4ad3b141"}
Nov 02 02:09:09.497515 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.496684    5639 scope.go:122] "RemoveContainer" containerID="4cc52e35c4e0543b16494f86403df9c229aeea264497edd5c8cf3b296e658725"
Nov 02 02:09:09.509385 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.509363    5639 scope.go:122] "RemoveContainer" containerID="4cc52e35c4e0543b16494f86403df9c229aeea264497edd5c8cf3b296e658725"
Nov 02 02:09:09.509666 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:09:09.509647    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cc52e35c4e0543b16494f86403df9c229aeea264497edd5c8cf3b296e658725\": container with ID starting with 4cc52e35c4e0543b16494f86403df9c229aeea264497edd5c8cf3b296e658725 not found: ID does not exist" containerID="4cc52e35c4e0543b16494f86403df9c229aeea264497edd5c8cf3b296e658725"
Nov 02 02:09:09.509716 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.509676    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc52e35c4e0543b16494f86403df9c229aeea264497edd5c8cf3b296e658725"} err="failed to get container status \"4cc52e35c4e0543b16494f86403df9c229aeea264497edd5c8cf3b296e658725\": rpc error: code = NotFound desc = could not find container \"4cc52e35c4e0543b16494f86403df9c229aeea264497edd5c8cf3b296e658725\": container with ID starting with 4cc52e35c4e0543b16494f86403df9c229aeea264497edd5c8cf3b296e658725 not found: ID does not exist"
Nov 02 02:09:09.662584 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.662547    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"]
Nov 02 02:09:09.668228 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:09.668197    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-mqr6f"]
Nov 02 02:09:11.220369 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:11.220326    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4443617-f4c0-40c4-a3bf-0b6241d6ce97" path="/var/lib/kubelet/pods/c4443617-f4c0-40c4-a3bf-0b6241d6ce97/volumes"
Nov 02 02:09:20.317771 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:20.317688    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-76e530-wcs89"]
Nov 02 02:09:20.366620 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:20.366580    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-76e530-wcs89"]
Nov 02 02:09:21.218605 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:21.218569    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8c363fc-bfe1-4102-9250-971837638835" path="/var/lib/kubelet/pods/c8c363fc-bfe1-4102-9250-971837638835/volumes"
Nov 02 02:09:25.190931 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:25.190895    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:09:27.091998 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:27.091963    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"]
Nov 02 02:09:27.092473 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:27.092214    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w" podUID="7b64c8f8-821f-4d7b-a684-40433fb5396c" containerName="istio-proxy" containerID="cri-o://662c817bc9544e127aa4e2cce28f109003a29057122dbf5031ede4cdf4e1f7f0" gracePeriod=30
Nov 02 02:09:27.424520 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:09:27.424439    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:09:27.962776 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:27.962739    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9039bf-pg48g"]
Nov 02 02:09:27.963343 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:27.963325    5639 cpu_manager.go:414] "containerMap: removing container" podUID="a777f857-bab6-465f-b718-d3d9b2d22716" containerName="revoke-keys"
Nov 02 02:09:27.963440 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:27.963345    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a777f857-bab6-465f-b718-d3d9b2d22716" containerName="revoke-keys"
Nov 02 02:09:27.963440 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:27.963367    5639 cpu_manager.go:414] "containerMap: removing container" podUID="4b5f60d5-7e21-48f5-96fb-ad1c519bfae0" containerName="payload-processing"
Nov 02 02:09:27.963440 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:27.963373    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b5f60d5-7e21-48f5-96fb-ad1c519bfae0" containerName="payload-processing"
Nov 02 02:09:27.963440 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:27.963380    5639 cpu_manager.go:414] "containerMap: removing container" podUID="c4443617-f4c0-40c4-a3bf-0b6241d6ce97" containerName="istio-proxy"
Nov 02 02:09:27.963440 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:27.963385    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4443617-f4c0-40c4-a3bf-0b6241d6ce97" containerName="istio-proxy"
Nov 02 02:09:27.963440 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:27.963395    5639 cpu_manager.go:414] "containerMap: removing container" podUID="dae611da-8694-4348-9dff-90bd553e95ec" containerName="istio-proxy"
Nov 02 02:09:27.963440 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:27.963399    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae611da-8694-4348-9dff-90bd553e95ec" containerName="istio-proxy"
Nov 02 02:09:27.963440 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:27.963427    5639 cpu_manager.go:414] "containerMap: removing container" podUID="2da5e48a-b3fd-4b3a-97d9-505aea24dc94" containerName="istio-proxy"
Nov 02 02:09:27.963440 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:27.963432    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da5e48a-b3fd-4b3a-97d9-505aea24dc94" containerName="istio-proxy"
Nov 02 02:09:27.963908 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:27.963518    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da5e48a-b3fd-4b3a-97d9-505aea24dc94" containerName="istio-proxy"
Nov 02 02:09:27.963908 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:27.963527    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="a777f857-bab6-465f-b718-d3d9b2d22716" containerName="revoke-keys"
Nov 02 02:09:27.963908 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:27.963536    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b5f60d5-7e21-48f5-96fb-ad1c519bfae0" containerName="payload-processing"
Nov 02 02:09:27.963908 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:27.963541    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4443617-f4c0-40c4-a3bf-0b6241d6ce97" containerName="istio-proxy"
Nov 02 02:09:27.963908 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:27.963547    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae611da-8694-4348-9dff-90bd553e95ec" containerName="istio-proxy"
Nov 02 02:09:27.963908 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:27.963555    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae611da-8694-4348-9dff-90bd553e95ec" containerName="istio-proxy"
Nov 02 02:09:27.968167 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:27.968144    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9039bf-pg48g"
Nov 02 02:09:28.024022 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:28.023991    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9039bf-pg48g"]
Nov 02 02:09:28.035110 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:28.035078    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4aac84c-d970-4ec4-bcea-6c24f59a67bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9039bf-pg48g\" (UID: \"a4aac84c-d970-4ec4-bcea-6c24f59a67bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9039bf-pg48g"
Nov 02 02:09:28.035305 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:28.035146    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znbw9\" (UniqueName: \"kubernetes.io/projected/a4aac84c-d970-4ec4-bcea-6c24f59a67bd-kube-api-access-znbw9\") pod \"maas-api-revoke-keys-e2e-api-b-9039bf-pg48g\" (UID: \"a4aac84c-d970-4ec4-bcea-6c24f59a67bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9039bf-pg48g"
Nov 02 02:09:28.035305 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:28.035262    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4aac84c-d970-4ec4-bcea-6c24f59a67bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9039bf-pg48g\" (UID: \"a4aac84c-d970-4ec4-bcea-6c24f59a67bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9039bf-pg48g"
Nov 02 02:09:28.069996 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:28.069960    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:09:28.070165 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:28.070006    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:09:28.070165 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:28.070044    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:09:28.070165 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:28.070066    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:09:28.136578 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:28.136543    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4aac84c-d970-4ec4-bcea-6c24f59a67bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9039bf-pg48g\" (UID: \"a4aac84c-d970-4ec4-bcea-6c24f59a67bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9039bf-pg48g"
Nov 02 02:09:28.137062 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:28.136603    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znbw9\" (UniqueName: \"kubernetes.io/projected/a4aac84c-d970-4ec4-bcea-6c24f59a67bd-kube-api-access-znbw9\") pod \"maas-api-revoke-keys-e2e-api-b-9039bf-pg48g\" (UID: \"a4aac84c-d970-4ec4-bcea-6c24f59a67bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9039bf-pg48g"
Nov 02 02:09:28.137062 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:28.136631    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4aac84c-d970-4ec4-bcea-6c24f59a67bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9039bf-pg48g\" (UID: \"a4aac84c-d970-4ec4-bcea-6c24f59a67bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9039bf-pg48g"
Nov 02 02:09:28.137377 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:28.137357    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4aac84c-d970-4ec4-bcea-6c24f59a67bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9039bf-pg48g\" (UID: \"a4aac84c-d970-4ec4-bcea-6c24f59a67bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9039bf-pg48g"
Nov 02 02:09:28.139296 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:28.139259    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4aac84c-d970-4ec4-bcea-6c24f59a67bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9039bf-pg48g\" (UID: \"a4aac84c-d970-4ec4-bcea-6c24f59a67bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9039bf-pg48g"
Nov 02 02:09:28.198304 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:28.198254    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znbw9\" (UniqueName: \"kubernetes.io/projected/a4aac84c-d970-4ec4-bcea-6c24f59a67bd-kube-api-access-znbw9\") pod \"maas-api-revoke-keys-e2e-api-b-9039bf-pg48g\" (UID: \"a4aac84c-d970-4ec4-bcea-6c24f59a67bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9039bf-pg48g"
Nov 02 02:09:28.283139 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:28.283052    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9039bf-pg48g"
Nov 02 02:09:28.661022 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:28.660994    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9039bf-pg48g"]
Nov 02 02:09:28.663105 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:09:28.663075    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4aac84c_d970_4ec4_bcea_6c24f59a67bd.slice/crio-4d80fb382586546e606cfa1d37c26d150e36ae32f8107e00c0b87039747ac582 WatchSource:0}: Error finding container 4d80fb382586546e606cfa1d37c26d150e36ae32f8107e00c0b87039747ac582: Status 404 returned error can't find the container with id 4d80fb382586546e606cfa1d37c26d150e36ae32f8107e00c0b87039747ac582
Nov 02 02:09:29.644394 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:29.644354    5639 generic.go:356] "Generic (PLEG): container finished" podID="a4aac84c-d970-4ec4-bcea-6c24f59a67bd" containerID="77c18ec06259d9e36f6ad29bd63375b97eb790a22e0998d9a57c7c479fb4595b" exitCode=0
Nov 02 02:09:29.644798 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:29.644446    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9039bf-pg48g" event={"ID":"a4aac84c-d970-4ec4-bcea-6c24f59a67bd","Type":"ContainerDied","Data":"77c18ec06259d9e36f6ad29bd63375b97eb790a22e0998d9a57c7c479fb4595b"}
Nov 02 02:09:29.644798 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:29.644508    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9039bf-pg48g" event={"ID":"a4aac84c-d970-4ec4-bcea-6c24f59a67bd","Type":"ContainerStarted","Data":"4d80fb382586546e606cfa1d37c26d150e36ae32f8107e00c0b87039747ac582"}
Nov 02 02:09:30.813792 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:30.813752    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9039bf-pg48g"
Nov 02 02:09:30.870890 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:30.870856    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4aac84c-d970-4ec4-bcea-6c24f59a67bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4aac84c-d970-4ec4-bcea-6c24f59a67bd-maas-api-openshift-ca\") pod \"a4aac84c-d970-4ec4-bcea-6c24f59a67bd\" (UID: \"a4aac84c-d970-4ec4-bcea-6c24f59a67bd\") "
Nov 02 02:09:30.871096 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:30.870972    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4aac84c-d970-4ec4-bcea-6c24f59a67bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4aac84c-d970-4ec4-bcea-6c24f59a67bd-maas-api-xks-ca\") pod \"a4aac84c-d970-4ec4-bcea-6c24f59a67bd\" (UID: \"a4aac84c-d970-4ec4-bcea-6c24f59a67bd\") "
Nov 02 02:09:30.871096 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:30.871044    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4aac84c-d970-4ec4-bcea-6c24f59a67bd-kube-api-access-znbw9\" (UniqueName: \"kubernetes.io/projected/a4aac84c-d970-4ec4-bcea-6c24f59a67bd-kube-api-access-znbw9\") pod \"a4aac84c-d970-4ec4-bcea-6c24f59a67bd\" (UID: \"a4aac84c-d970-4ec4-bcea-6c24f59a67bd\") "
Nov 02 02:09:30.871248 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:30.871219    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4aac84c-d970-4ec4-bcea-6c24f59a67bd-maas-api-openshift-ca" pod "a4aac84c-d970-4ec4-bcea-6c24f59a67bd" (UID: "a4aac84c-d970-4ec4-bcea-6c24f59a67bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:09:30.871557 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:30.871537    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4aac84c-d970-4ec4-bcea-6c24f59a67bd-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:30.873340 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:30.873312    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4aac84c-d970-4ec4-bcea-6c24f59a67bd-maas-api-xks-ca" pod "a4aac84c-d970-4ec4-bcea-6c24f59a67bd" (UID: "a4aac84c-d970-4ec4-bcea-6c24f59a67bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:09:30.873434 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:30.873381    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4aac84c-d970-4ec4-bcea-6c24f59a67bd-kube-api-access-znbw9" pod "a4aac84c-d970-4ec4-bcea-6c24f59a67bd" (UID: "a4aac84c-d970-4ec4-bcea-6c24f59a67bd"). InnerVolumeSpecName "kube-api-access-znbw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:30.973342 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:30.973214    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4aac84c-d970-4ec4-bcea-6c24f59a67bd-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:30.973342 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:30.973245    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znbw9\" (UniqueName: \"kubernetes.io/projected/a4aac84c-d970-4ec4-bcea-6c24f59a67bd-kube-api-access-znbw9\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:31.492939 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:31.492850    5639 patch_prober.go:29] interesting pod/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:09:31.492939 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:31.492911    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w" podUID="7b64c8f8-821f-4d7b-a684-40433fb5396c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:09:31.660242 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:31.660207    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9039bf-pg48g"
Nov 02 02:09:31.660431 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:31.660242    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9039bf-pg48g" event={"ID":"a4aac84c-d970-4ec4-bcea-6c24f59a67bd","Type":"ContainerDied","Data":"4d80fb382586546e606cfa1d37c26d150e36ae32f8107e00c0b87039747ac582"}
Nov 02 02:09:31.660431 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:31.660306    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d80fb382586546e606cfa1d37c26d150e36ae32f8107e00c0b87039747ac582"
Nov 02 02:09:32.373700 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.373670    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:09:32.492893 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.492862    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-workload-certs\") pod \"7b64c8f8-821f-4d7b-a684-40433fb5396c\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") "
Nov 02 02:09:32.492893 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.492894    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-credential-socket\") pod \"7b64c8f8-821f-4d7b-a684-40433fb5396c\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") "
Nov 02 02:09:32.493135 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.492923    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b64c8f8-821f-4d7b-a684-40433fb5396c-kube-api-access-w6mks\" (UniqueName: \"kubernetes.io/projected/7b64c8f8-821f-4d7b-a684-40433fb5396c-kube-api-access-w6mks\") pod \"7b64c8f8-821f-4d7b-a684-40433fb5396c\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") "
Nov 02 02:09:32.493135 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.492970    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-data\") pod \"7b64c8f8-821f-4d7b-a684-40433fb5396c\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") "
Nov 02 02:09:32.493135 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.493012    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b64c8f8-821f-4d7b-a684-40433fb5396c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b64c8f8-821f-4d7b-a684-40433fb5396c-istiod-ca-cert\") pod \"7b64c8f8-821f-4d7b-a684-40433fb5396c\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") "
Nov 02 02:09:32.493135 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.493045    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-podinfo\") pod \"7b64c8f8-821f-4d7b-a684-40433fb5396c\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") "
Nov 02 02:09:32.493135 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.493114    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-workload-socket\") pod \"7b64c8f8-821f-4d7b-a684-40433fb5396c\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") "
Nov 02 02:09:32.493459 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.493161    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-envoy\") pod \"7b64c8f8-821f-4d7b-a684-40433fb5396c\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") "
Nov 02 02:09:32.493459 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.493219    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-credential-socket" pod "7b64c8f8-821f-4d7b-a684-40433fb5396c" (UID: "7b64c8f8-821f-4d7b-a684-40433fb5396c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:32.493459 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.493253    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-token\" (UniqueName: \"kubernetes.io/projected/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-token\") pod \"7b64c8f8-821f-4d7b-a684-40433fb5396c\" (UID: \"7b64c8f8-821f-4d7b-a684-40433fb5396c\") "
Nov 02 02:09:32.493459 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.493252    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-workload-certs" pod "7b64c8f8-821f-4d7b-a684-40433fb5396c" (UID: "7b64c8f8-821f-4d7b-a684-40433fb5396c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:32.493684 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.493489    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-workload-socket" pod "7b64c8f8-821f-4d7b-a684-40433fb5396c" (UID: "7b64c8f8-821f-4d7b-a684-40433fb5396c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:32.493684 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.493606    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-data" pod "7b64c8f8-821f-4d7b-a684-40433fb5396c" (UID: "7b64c8f8-821f-4d7b-a684-40433fb5396c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:32.493684 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.493627    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b64c8f8-821f-4d7b-a684-40433fb5396c-istiod-ca-cert" pod "7b64c8f8-821f-4d7b-a684-40433fb5396c" (UID: "7b64c8f8-821f-4d7b-a684-40433fb5396c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:09:32.493932 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.493910    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:32.494007 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.493942    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b64c8f8-821f-4d7b-a684-40433fb5396c-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:32.494007 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.494001    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:32.494314 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.494047    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:32.494314 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.494104    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:32.495747 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.495722    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-envoy" pod "7b64c8f8-821f-4d7b-a684-40433fb5396c" (UID: "7b64c8f8-821f-4d7b-a684-40433fb5396c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:32.495912 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.495892    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-podinfo" pod "7b64c8f8-821f-4d7b-a684-40433fb5396c" (UID: "7b64c8f8-821f-4d7b-a684-40433fb5396c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:09:32.495912 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.495895    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b64c8f8-821f-4d7b-a684-40433fb5396c-kube-api-access-w6mks" pod "7b64c8f8-821f-4d7b-a684-40433fb5396c" (UID: "7b64c8f8-821f-4d7b-a684-40433fb5396c"). InnerVolumeSpecName "kube-api-access-w6mks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:32.496012 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.495998    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-token" pod "7b64c8f8-821f-4d7b-a684-40433fb5396c" (UID: "7b64c8f8-821f-4d7b-a684-40433fb5396c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:32.595258 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.595222    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:32.595258 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.595250    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:32.595258 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.595268    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b64c8f8-821f-4d7b-a684-40433fb5396c-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:32.595653 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.595294    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6mks\" (UniqueName: \"kubernetes.io/projected/7b64c8f8-821f-4d7b-a684-40433fb5396c-kube-api-access-w6mks\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:32.670314 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.670191    5639 generic.go:356] "Generic (PLEG): container finished" podID="7b64c8f8-821f-4d7b-a684-40433fb5396c" containerID="662c817bc9544e127aa4e2cce28f109003a29057122dbf5031ede4cdf4e1f7f0" exitCode=0
Nov 02 02:09:32.670314 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.670238    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"
Nov 02 02:09:32.670314 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.670267    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w" event={"ID":"7b64c8f8-821f-4d7b-a684-40433fb5396c","Type":"ContainerDied","Data":"662c817bc9544e127aa4e2cce28f109003a29057122dbf5031ede4cdf4e1f7f0"}
Nov 02 02:09:32.670584 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.670347    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w" event={"ID":"7b64c8f8-821f-4d7b-a684-40433fb5396c","Type":"ContainerDied","Data":"9d49e618bb15632edb3890edbad06f7a43e4042436eaf531cf756ddb49cdd5b3"}
Nov 02 02:09:32.670584 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.670379    5639 scope.go:122] "RemoveContainer" containerID="662c817bc9544e127aa4e2cce28f109003a29057122dbf5031ede4cdf4e1f7f0"
Nov 02 02:09:32.681748 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.681729    5639 scope.go:122] "RemoveContainer" containerID="ecf06b1e5351a646431263dc195d597b322c9baee00af15d3f8288fa6b82f872"
Nov 02 02:09:32.692601 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.692576    5639 scope.go:122] "RemoveContainer" containerID="662c817bc9544e127aa4e2cce28f109003a29057122dbf5031ede4cdf4e1f7f0"
Nov 02 02:09:32.692930 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:09:32.692908    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"662c817bc9544e127aa4e2cce28f109003a29057122dbf5031ede4cdf4e1f7f0\": container with ID starting with 662c817bc9544e127aa4e2cce28f109003a29057122dbf5031ede4cdf4e1f7f0 not found: ID does not exist" containerID="662c817bc9544e127aa4e2cce28f109003a29057122dbf5031ede4cdf4e1f7f0"
Nov 02 02:09:32.693014 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.692944    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"662c817bc9544e127aa4e2cce28f109003a29057122dbf5031ede4cdf4e1f7f0"} err="failed to get container status \"662c817bc9544e127aa4e2cce28f109003a29057122dbf5031ede4cdf4e1f7f0\": rpc error: code = NotFound desc = could not find container \"662c817bc9544e127aa4e2cce28f109003a29057122dbf5031ede4cdf4e1f7f0\": container with ID starting with 662c817bc9544e127aa4e2cce28f109003a29057122dbf5031ede4cdf4e1f7f0 not found: ID does not exist"
Nov 02 02:09:32.693014 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.692970    5639 scope.go:122] "RemoveContainer" containerID="ecf06b1e5351a646431263dc195d597b322c9baee00af15d3f8288fa6b82f872"
Nov 02 02:09:32.693219 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:09:32.693201    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecf06b1e5351a646431263dc195d597b322c9baee00af15d3f8288fa6b82f872\": container with ID starting with ecf06b1e5351a646431263dc195d597b322c9baee00af15d3f8288fa6b82f872 not found: ID does not exist" containerID="ecf06b1e5351a646431263dc195d597b322c9baee00af15d3f8288fa6b82f872"
Nov 02 02:09:32.693260 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.693227    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecf06b1e5351a646431263dc195d597b322c9baee00af15d3f8288fa6b82f872"} err="failed to get container status \"ecf06b1e5351a646431263dc195d597b322c9baee00af15d3f8288fa6b82f872\": rpc error: code = NotFound desc = could not find container \"ecf06b1e5351a646431263dc195d597b322c9baee00af15d3f8288fa6b82f872\": container with ID starting with ecf06b1e5351a646431263dc195d597b322c9baee00af15d3f8288fa6b82f872 not found: ID does not exist"
Nov 02 02:09:32.836556 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.836521    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"]
Nov 02 02:09:32.840861 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:32.840827    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-76e530-openshift-default-6c5557fcc-rxt5w"]
Nov 02 02:09:33.218763 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:33.218730    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b64c8f8-821f-4d7b-a684-40433fb5396c" path="/var/lib/kubelet/pods/7b64c8f8-821f-4d7b-a684-40433fb5396c/volumes"
Nov 02 02:09:38.725392 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:38.725359    5639 scope.go:122] "RemoveContainer" containerID="7c83f63d3e104b328d502afc2764281cc369479646c0fa7465f436970ec1f6ea"
Nov 02 02:09:38.735536 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:38.735512    5639 scope.go:122] "RemoveContainer" containerID="faca89117f02a1ab103ac1e1a4e12e76341420fa731322997cf79dd94de9ca1e"
Nov 02 02:09:38.745048 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:38.745024    5639 scope.go:122] "RemoveContainer" containerID="9e3b83f00646fa88eb93d86ecc55f1a7ba104129393bd0eb903d945c4398b24d"
Nov 02 02:09:38.757570 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:38.757543    5639 scope.go:122] "RemoveContainer" containerID="e14ba01c82cc1a65227cfb6dc775e592f39e17f9873c3d54e3b90e0c84b516f2"
Nov 02 02:09:38.767137 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:38.767113    5639 scope.go:122] "RemoveContainer" containerID="d749c0c7b223a006350800493d57af6b3aa98dc8a0aa525db35b41b81297a0da"
Nov 02 02:09:38.776493 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:38.776469    5639 scope.go:122] "RemoveContainer" containerID="38800407ce79e559141cff532f0255c4521407dd2de4f5d8f80da53652a51df0"
Nov 02 02:09:39.555572 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:39.555535    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66"]
Nov 02 02:09:40.093941 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.093903    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"]
Nov 02 02:09:40.094435 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.094171    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz" podUID="604ec2ee-bfbc-4ba6-aa7f-0e61310b5563" containerName="discovery" containerID="cri-o://25a7d87c9b08321d64b35f27358c936d387c5407dff592e65f8444d2d8636db1" gracePeriod=30
Nov 02 02:09:40.367800 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.367773    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 02:09:40.493837 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.493804    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-csr-dns-cert\") pod \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") "
Nov 02 02:09:40.493837 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.493839    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-csr-ca-configmap\") pod \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") "
Nov 02 02:09:40.494098 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.493884    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-local-certs\") pod \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") "
Nov 02 02:09:40.494098 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.494017    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-kube-api-access-lh5l8\" (UniqueName: \"kubernetes.io/projected/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-kube-api-access-lh5l8\") pod \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") "
Nov 02 02:09:40.494098 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.494050    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-token\" (UniqueName: \"kubernetes.io/projected/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-token\") pod \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") "
Nov 02 02:09:40.494368 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.494108    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-cacerts\" (UniqueName: \"kubernetes.io/secret/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-cacerts\") pod \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") "
Nov 02 02:09:40.494368 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.494318    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-csr-ca-configmap" pod "604ec2ee-bfbc-4ba6-aa7f-0e61310b5563" (UID: "604ec2ee-bfbc-4ba6-aa7f-0e61310b5563"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:09:40.494671 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.494647    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-kubeconfig\") pod \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\" (UID: \"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563\") "
Nov 02 02:09:40.496194 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.496171    5639 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-csr-ca-configmap\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:40.497490 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.497045    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-kube-api-access-lh5l8" pod "604ec2ee-bfbc-4ba6-aa7f-0e61310b5563" (UID: "604ec2ee-bfbc-4ba6-aa7f-0e61310b5563"). InnerVolumeSpecName "kube-api-access-lh5l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:40.497490 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.497143    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-local-certs" pod "604ec2ee-bfbc-4ba6-aa7f-0e61310b5563" (UID: "604ec2ee-bfbc-4ba6-aa7f-0e61310b5563"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:40.497490 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.497137    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-cacerts" pod "604ec2ee-bfbc-4ba6-aa7f-0e61310b5563" (UID: "604ec2ee-bfbc-4ba6-aa7f-0e61310b5563"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:09:40.497869 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.497839    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-kubeconfig" pod "604ec2ee-bfbc-4ba6-aa7f-0e61310b5563" (UID: "604ec2ee-bfbc-4ba6-aa7f-0e61310b5563"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:09:40.497977 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.497948    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-csr-dns-cert" pod "604ec2ee-bfbc-4ba6-aa7f-0e61310b5563" (UID: "604ec2ee-bfbc-4ba6-aa7f-0e61310b5563"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:09:40.498923 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.498902    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-token" pod "604ec2ee-bfbc-4ba6-aa7f-0e61310b5563" (UID: "604ec2ee-bfbc-4ba6-aa7f-0e61310b5563"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:40.598442 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.598206    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh5l8\" (UniqueName: \"kubernetes.io/projected/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-kube-api-access-lh5l8\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:40.598442 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.598249    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:40.598442 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.598265    5639 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-cacerts\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:40.598442 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.598316    5639 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-kubeconfig\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:40.598442 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.598333    5639 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-istio-csr-dns-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:40.598442 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.598348    5639 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563-local-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:40.733629 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.733596    5639 generic.go:356] "Generic (PLEG): container finished" podID="604ec2ee-bfbc-4ba6-aa7f-0e61310b5563" containerID="25a7d87c9b08321d64b35f27358c936d387c5407dff592e65f8444d2d8636db1" exitCode=0
Nov 02 02:09:40.733815 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.733651    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"
Nov 02 02:09:40.733815 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.733675    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz" event={"ID":"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563","Type":"ContainerDied","Data":"25a7d87c9b08321d64b35f27358c936d387c5407dff592e65f8444d2d8636db1"}
Nov 02 02:09:40.733815 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.733734    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz" event={"ID":"604ec2ee-bfbc-4ba6-aa7f-0e61310b5563","Type":"ContainerDied","Data":"864aa1eca13c2a35e8e4699dd4f1ecdd8553ed0846550450fcb6ff2b3bf71180"}
Nov 02 02:09:40.733815 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.733762    5639 scope.go:122] "RemoveContainer" containerID="25a7d87c9b08321d64b35f27358c936d387c5407dff592e65f8444d2d8636db1"
Nov 02 02:09:40.747959 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.747934    5639 scope.go:122] "RemoveContainer" containerID="25a7d87c9b08321d64b35f27358c936d387c5407dff592e65f8444d2d8636db1"
Nov 02 02:09:40.748298 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:09:40.748260    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25a7d87c9b08321d64b35f27358c936d387c5407dff592e65f8444d2d8636db1\": container with ID starting with 25a7d87c9b08321d64b35f27358c936d387c5407dff592e65f8444d2d8636db1 not found: ID does not exist" containerID="25a7d87c9b08321d64b35f27358c936d387c5407dff592e65f8444d2d8636db1"
Nov 02 02:09:40.748383 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.748321    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25a7d87c9b08321d64b35f27358c936d387c5407dff592e65f8444d2d8636db1"} err="failed to get container status \"25a7d87c9b08321d64b35f27358c936d387c5407dff592e65f8444d2d8636db1\": rpc error: code = NotFound desc = could not find container \"25a7d87c9b08321d64b35f27358c936d387c5407dff592e65f8444d2d8636db1\": container with ID starting with 25a7d87c9b08321d64b35f27358c936d387c5407dff592e65f8444d2d8636db1 not found: ID does not exist"
Nov 02 02:09:40.899484 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.899405    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"]
Nov 02 02:09:40.903991 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:40.903958    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qkrbz"]
Nov 02 02:09:41.220616 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:41.220537    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="604ec2ee-bfbc-4ba6-aa7f-0e61310b5563" path="/var/lib/kubelet/pods/604ec2ee-bfbc-4ba6-aa7f-0e61310b5563/volumes"
Nov 02 02:09:44.556001 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:44.555948    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66" podUID="264280bd-0869-4da4-a6d0-9d393849a85a" containerName="payload-pre-processing" containerID="cri-o://59f7492ccb0bd350681eaf06e6e05664b247d4ef76fbc61568c65722add6c484" gracePeriod=25
Nov 02 02:09:44.770214 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:44.770178    5639 generic.go:356] "Generic (PLEG): container finished" podID="264280bd-0869-4da4-a6d0-9d393849a85a" containerID="59f7492ccb0bd350681eaf06e6e05664b247d4ef76fbc61568c65722add6c484" exitCode=0
Nov 02 02:09:44.770467 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:44.770425    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66" event={"ID":"264280bd-0869-4da4-a6d0-9d393849a85a","Type":"ContainerDied","Data":"59f7492ccb0bd350681eaf06e6e05664b247d4ef76fbc61568c65722add6c484"}
Nov 02 02:09:44.826896 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:44.826812    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66"
Nov 02 02:09:44.844218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:44.844189    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/264280bd-0869-4da4-a6d0-9d393849a85a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/264280bd-0869-4da4-a6d0-9d393849a85a-plugins-config-volume\") pod \"264280bd-0869-4da4-a6d0-9d393849a85a\" (UID: \"264280bd-0869-4da4-a6d0-9d393849a85a\") "
Nov 02 02:09:44.844423 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:44.844229    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/264280bd-0869-4da4-a6d0-9d393849a85a-kube-api-access-48dg8\" (UniqueName: \"kubernetes.io/projected/264280bd-0869-4da4-a6d0-9d393849a85a-kube-api-access-48dg8\") pod \"264280bd-0869-4da4-a6d0-9d393849a85a\" (UID: \"264280bd-0869-4da4-a6d0-9d393849a85a\") "
Nov 02 02:09:44.844600 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:44.844579    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/264280bd-0869-4da4-a6d0-9d393849a85a-plugins-config-volume" pod "264280bd-0869-4da4-a6d0-9d393849a85a" (UID: "264280bd-0869-4da4-a6d0-9d393849a85a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:09:44.844688 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:44.844675    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/264280bd-0869-4da4-a6d0-9d393849a85a-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:44.846473 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:44.846451    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/264280bd-0869-4da4-a6d0-9d393849a85a-kube-api-access-48dg8" pod "264280bd-0869-4da4-a6d0-9d393849a85a" (UID: "264280bd-0869-4da4-a6d0-9d393849a85a"). InnerVolumeSpecName "kube-api-access-48dg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:44.945593 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:44.945564    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48dg8\" (UniqueName: \"kubernetes.io/projected/264280bd-0869-4da4-a6d0-9d393849a85a-kube-api-access-48dg8\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:45.779746 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:45.779714    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66"
Nov 02 02:09:45.780200 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:45.779744    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66" event={"ID":"264280bd-0869-4da4-a6d0-9d393849a85a","Type":"ContainerDied","Data":"c4ec928c84b74c4105738a491598815321a6c7748f948a9ff268a6d209c2d415"}
Nov 02 02:09:45.780200 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:45.779809    5639 scope.go:122] "RemoveContainer" containerID="59f7492ccb0bd350681eaf06e6e05664b247d4ef76fbc61568c65722add6c484"
Nov 02 02:09:45.945718 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:45.945682    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66"]
Nov 02 02:09:45.995340 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:45.995298    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9039bf-586db859dc-mxj66"]
Nov 02 02:09:47.219645 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:47.219605    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="264280bd-0869-4da4-a6d0-9d393849a85a" path="/var/lib/kubelet/pods/264280bd-0869-4da4-a6d0-9d393849a85a/volumes"
Nov 02 02:09:47.684015 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:47.683979    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:09:48.055310 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:48.055194    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-143b1c14-n5mks"]
Nov 02 02:09:48.060678 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:48.060653    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc2929a0-jdt45"]
Nov 02 02:09:48.120540 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:48.116052    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-143b1c14-n5mks"]
Nov 02 02:09:48.120540 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:48.118610    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc2929a0-jdt45"]
Nov 02 02:09:49.219422 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:49.219384    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a93f67ee-ce71-45c5-a7ec-e04378559fe2" path="/var/lib/kubelet/pods/a93f67ee-ce71-45c5-a7ec-e04378559fe2/volumes"
Nov 02 02:09:49.219836 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:09:49.219821    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6d5428f-ff79-4948-8a60-8dac2796d29f" path="/var/lib/kubelet/pods/c6d5428f-ff79-4948-8a60-8dac2796d29f/volumes"
Nov 02 02:09:57.421002 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:09:57.420957    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:10:24.350778 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:24.350741    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c78625-wch9w"]
Nov 02 02:10:24.400275 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:24.400244    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c78625-wch9w"]
Nov 02 02:10:25.221197 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:25.221160    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a777f857-bab6-465f-b718-d3d9b2d22716" path="/var/lib/kubelet/pods/a777f857-bab6-465f-b718-d3d9b2d22716/volumes"
Nov 02 02:10:27.422844 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:10:27.422804    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:10:32.045826 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.045787    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff"]
Nov 02 02:10:32.046366 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.046344    5639 cpu_manager.go:414] "containerMap: removing container" podUID="604ec2ee-bfbc-4ba6-aa7f-0e61310b5563" containerName="discovery"
Nov 02 02:10:32.046426 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.046368    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604ec2ee-bfbc-4ba6-aa7f-0e61310b5563" containerName="discovery"
Nov 02 02:10:32.046426 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.046381    5639 cpu_manager.go:414] "containerMap: removing container" podUID="264280bd-0869-4da4-a6d0-9d393849a85a" containerName="payload-pre-processing"
Nov 02 02:10:32.046426 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.046386    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="264280bd-0869-4da4-a6d0-9d393849a85a" containerName="payload-pre-processing"
Nov 02 02:10:32.046426 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.046397    5639 cpu_manager.go:414] "containerMap: removing container" podUID="dae611da-8694-4348-9dff-90bd553e95ec" containerName="istio-proxy"
Nov 02 02:10:32.046426 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.046402    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae611da-8694-4348-9dff-90bd553e95ec" containerName="istio-proxy"
Nov 02 02:10:32.046426 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.046419    5639 cpu_manager.go:414] "containerMap: removing container" podUID="7b64c8f8-821f-4d7b-a684-40433fb5396c" containerName="istio-proxy"
Nov 02 02:10:32.046426 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.046424    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b64c8f8-821f-4d7b-a684-40433fb5396c" containerName="istio-proxy"
Nov 02 02:10:32.046653 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.046433    5639 cpu_manager.go:414] "containerMap: removing container" podUID="a4aac84c-d970-4ec4-bcea-6c24f59a67bd" containerName="revoke-keys"
Nov 02 02:10:32.046653 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.046439    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4aac84c-d970-4ec4-bcea-6c24f59a67bd" containerName="revoke-keys"
Nov 02 02:10:32.046653 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.046538    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="604ec2ee-bfbc-4ba6-aa7f-0e61310b5563" containerName="discovery"
Nov 02 02:10:32.046653 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.046547    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="264280bd-0869-4da4-a6d0-9d393849a85a" containerName="payload-pre-processing"
Nov 02 02:10:32.046653 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.046558    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b64c8f8-821f-4d7b-a684-40433fb5396c" containerName="istio-proxy"
Nov 02 02:10:32.046653 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.046564    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4aac84c-d970-4ec4-bcea-6c24f59a67bd" containerName="revoke-keys"
Nov 02 02:10:32.049819 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.049801    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff"
Nov 02 02:10:32.108408 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.108374    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff"]
Nov 02 02:10:32.154635 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.154599    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:10:32.154635 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.154628    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:10:32.154635 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.154628    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:10:32.154969 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.154633    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:10:32.215505 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.215474    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add349ba-4927-476d-ab88-5336942c592f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff\" (UID: \"add349ba-4927-476d-ab88-5336942c592f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff"
Nov 02 02:10:32.215708 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.215528    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add349ba-4927-476d-ab88-5336942c592f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff\" (UID: \"add349ba-4927-476d-ab88-5336942c592f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff"
Nov 02 02:10:32.215708 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.215640    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf6zg\" (UniqueName: \"kubernetes.io/projected/add349ba-4927-476d-ab88-5336942c592f-kube-api-access-kf6zg\") pod \"maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff\" (UID: \"add349ba-4927-476d-ab88-5336942c592f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff"
Nov 02 02:10:32.317455 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.317354    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf6zg\" (UniqueName: \"kubernetes.io/projected/add349ba-4927-476d-ab88-5336942c592f-kube-api-access-kf6zg\") pod \"maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff\" (UID: \"add349ba-4927-476d-ab88-5336942c592f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff"
Nov 02 02:10:32.317758 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.317724    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add349ba-4927-476d-ab88-5336942c592f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff\" (UID: \"add349ba-4927-476d-ab88-5336942c592f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff"
Nov 02 02:10:32.317895 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.317869    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add349ba-4927-476d-ab88-5336942c592f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff\" (UID: \"add349ba-4927-476d-ab88-5336942c592f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff"
Nov 02 02:10:32.318644 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.318620    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add349ba-4927-476d-ab88-5336942c592f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff\" (UID: \"add349ba-4927-476d-ab88-5336942c592f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff"
Nov 02 02:10:32.320547 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.320523    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add349ba-4927-476d-ab88-5336942c592f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff\" (UID: \"add349ba-4927-476d-ab88-5336942c592f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff"
Nov 02 02:10:32.378684 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.378648    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf6zg\" (UniqueName: \"kubernetes.io/projected/add349ba-4927-476d-ab88-5336942c592f-kube-api-access-kf6zg\") pod \"maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff\" (UID: \"add349ba-4927-476d-ab88-5336942c592f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff"
Nov 02 02:10:32.663509 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:32.663471    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff"
Nov 02 02:10:33.041762 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:33.041732    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff"]
Nov 02 02:10:33.044188 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:10:33.044156    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd349ba_4927_476d_ab88_5336942c592f.slice/crio-b8e084cfccfe379c34a98cefb9a1a76ee79aa2eff31ce65037e833b0405db050 WatchSource:0}: Error finding container b8e084cfccfe379c34a98cefb9a1a76ee79aa2eff31ce65037e833b0405db050: Status 404 returned error can't find the container with id b8e084cfccfe379c34a98cefb9a1a76ee79aa2eff31ce65037e833b0405db050
Nov 02 02:10:33.096685 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:33.096649    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff" event={"ID":"add349ba-4927-476d-ab88-5336942c592f","Type":"ContainerStarted","Data":"b8e084cfccfe379c34a98cefb9a1a76ee79aa2eff31ce65037e833b0405db050"}
Nov 02 02:10:34.105006 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:34.104966    5639 generic.go:356] "Generic (PLEG): container finished" podID="add349ba-4927-476d-ab88-5336942c592f" containerID="f01cf1fb575f62d8f81b4be52e253eb797b54e9fd6a2dd74408accb0422191b1" exitCode=0
Nov 02 02:10:34.105555 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:34.105017    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff" event={"ID":"add349ba-4927-476d-ab88-5336942c592f","Type":"ContainerDied","Data":"f01cf1fb575f62d8f81b4be52e253eb797b54e9fd6a2dd74408accb0422191b1"}
Nov 02 02:10:35.259271 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:35.259241    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff"
Nov 02 02:10:35.359513 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:35.359481    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/add349ba-4927-476d-ab88-5336942c592f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add349ba-4927-476d-ab88-5336942c592f-maas-api-xks-ca\") pod \"add349ba-4927-476d-ab88-5336942c592f\" (UID: \"add349ba-4927-476d-ab88-5336942c592f\") "
Nov 02 02:10:35.359513 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:35.359520    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add349ba-4927-476d-ab88-5336942c592f-kube-api-access-kf6zg\" (UniqueName: \"kubernetes.io/projected/add349ba-4927-476d-ab88-5336942c592f-kube-api-access-kf6zg\") pod \"add349ba-4927-476d-ab88-5336942c592f\" (UID: \"add349ba-4927-476d-ab88-5336942c592f\") "
Nov 02 02:10:35.359779 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:35.359555    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/add349ba-4927-476d-ab88-5336942c592f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add349ba-4927-476d-ab88-5336942c592f-maas-api-openshift-ca\") pod \"add349ba-4927-476d-ab88-5336942c592f\" (UID: \"add349ba-4927-476d-ab88-5336942c592f\") "
Nov 02 02:10:35.359993 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:35.359963    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add349ba-4927-476d-ab88-5336942c592f-maas-api-openshift-ca" pod "add349ba-4927-476d-ab88-5336942c592f" (UID: "add349ba-4927-476d-ab88-5336942c592f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:10:35.360410 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:35.360311    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add349ba-4927-476d-ab88-5336942c592f-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:35.361886 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:35.361854    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add349ba-4927-476d-ab88-5336942c592f-kube-api-access-kf6zg" pod "add349ba-4927-476d-ab88-5336942c592f" (UID: "add349ba-4927-476d-ab88-5336942c592f"). InnerVolumeSpecName "kube-api-access-kf6zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:10:35.362333 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:35.362260    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/add349ba-4927-476d-ab88-5336942c592f-maas-api-xks-ca" pod "add349ba-4927-476d-ab88-5336942c592f" (UID: "add349ba-4927-476d-ab88-5336942c592f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:10:35.461873 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:35.461831    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add349ba-4927-476d-ab88-5336942c592f-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:35.461873 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:35.461867    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf6zg\" (UniqueName: \"kubernetes.io/projected/add349ba-4927-476d-ab88-5336942c592f-kube-api-access-kf6zg\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:36.119008 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:36.118973    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff"
Nov 02 02:10:36.119207 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:36.119014    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff" event={"ID":"add349ba-4927-476d-ab88-5336942c592f","Type":"ContainerDied","Data":"b8e084cfccfe379c34a98cefb9a1a76ee79aa2eff31ce65037e833b0405db050"}
Nov 02 02:10:36.119207 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:36.119065    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8e084cfccfe379c34a98cefb9a1a76ee79aa2eff31ce65037e833b0405db050"
Nov 02 02:10:36.499576 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:36.499486    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"]
Nov 02 02:10:36.507813 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:36.507781    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"]
Nov 02 02:10:38.825562 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:38.825504    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:10:38.843221 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:38.843186    5639 scope.go:122] "RemoveContainer" containerID="ddaa730d22f71270f7d7343088ca61c1ec8663472d1a1d7a6e1660ec7ebb04e7"
Nov 02 02:10:38.854038 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:38.854005    5639 scope.go:122] "RemoveContainer" containerID="86dd24898d4b287faf71ca7a899d3e5ab3884db504b06775707405819f4d9a55"
Nov 02 02:10:51.521165 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.521038    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main" containerID="cri-o://1eae94d5c379dfa2de7082f7beaa829d193398bae412fa0a567a99b565b26efe" gracePeriod=45
Nov 02 02:10:51.528140 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.528113    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main" containerID="cri-o://13427df22a25f684b69773ba0e4d9c76bd848aa0d1f0e56dfbb3f3b7298e5d37" gracePeriod=45
Nov 02 02:10:51.793121 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.793097    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 02:10:51.815296 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.815262    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 02:10:51.845425 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.845398    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-tmp-dir\") pod \"0dd080e9-d0c8-4341-ac96-562165899a80\" (UID: \"0dd080e9-d0c8-4341-ac96-562165899a80\") "
Nov 02 02:10:51.845648 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.845439    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-model-cache\") pod \"0dd080e9-d0c8-4341-ac96-562165899a80\" (UID: \"0dd080e9-d0c8-4341-ac96-562165899a80\") "
Nov 02 02:10:51.845648 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.845473    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-dshm\") pod \"0dd080e9-d0c8-4341-ac96-562165899a80\" (UID: \"0dd080e9-d0c8-4341-ac96-562165899a80\") "
Nov 02 02:10:51.845648 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.845613    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dd080e9-d0c8-4341-ac96-562165899a80-tls-certs\" (UniqueName: \"kubernetes.io/secret/0dd080e9-d0c8-4341-ac96-562165899a80-tls-certs\") pod \"0dd080e9-d0c8-4341-ac96-562165899a80\" (UID: \"0dd080e9-d0c8-4341-ac96-562165899a80\") "
Nov 02 02:10:51.845842 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.845689    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-tmp-dir" pod "0dd080e9-d0c8-4341-ac96-562165899a80" (UID: "0dd080e9-d0c8-4341-ac96-562165899a80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:51.845842 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.845728    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-home\" (UniqueName: \"kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-home\") pod \"0dd080e9-d0c8-4341-ac96-562165899a80\" (UID: \"0dd080e9-d0c8-4341-ac96-562165899a80\") "
Nov 02 02:10:51.845842 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.845772    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd080e9-d0c8-4341-ac96-562165899a80-kube-api-access-4c7r7\" (UniqueName: \"kubernetes.io/projected/0dd080e9-d0c8-4341-ac96-562165899a80-kube-api-access-4c7r7\") pod \"0dd080e9-d0c8-4341-ac96-562165899a80\" (UID: \"0dd080e9-d0c8-4341-ac96-562165899a80\") "
Nov 02 02:10:51.846085 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.846061    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-home" pod "0dd080e9-d0c8-4341-ac96-562165899a80" (UID: "0dd080e9-d0c8-4341-ac96-562165899a80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:51.846224 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.846205    5639 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-home\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:51.846362 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.846231    5639 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-tmp-dir\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:51.846362 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.846325    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-model-cache" pod "0dd080e9-d0c8-4341-ac96-562165899a80" (UID: "0dd080e9-d0c8-4341-ac96-562165899a80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:51.848549 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.848524    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-dshm" pod "0dd080e9-d0c8-4341-ac96-562165899a80" (UID: "0dd080e9-d0c8-4341-ac96-562165899a80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:51.848818 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.848791    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dd080e9-d0c8-4341-ac96-562165899a80-tls-certs" pod "0dd080e9-d0c8-4341-ac96-562165899a80" (UID: "0dd080e9-d0c8-4341-ac96-562165899a80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:10:51.848912 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.848831    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd080e9-d0c8-4341-ac96-562165899a80-kube-api-access-4c7r7" pod "0dd080e9-d0c8-4341-ac96-562165899a80" (UID: "0dd080e9-d0c8-4341-ac96-562165899a80"). InnerVolumeSpecName "kube-api-access-4c7r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:10:51.947086 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.947043    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-kube-api-access-bxftz\" (UniqueName: \"kubernetes.io/projected/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-kube-api-access-bxftz\") pod \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\" (UID: \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\") "
Nov 02 02:10:51.947263 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.947219    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-dshm\") pod \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\" (UID: \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\") "
Nov 02 02:10:51.947432 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.947266    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-home\" (UniqueName: \"kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-home\") pod \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\" (UID: \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\") "
Nov 02 02:10:51.947432 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.947321    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-model-cache\") pod \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\" (UID: \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\") "
Nov 02 02:10:51.947432 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.947390    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-tls-certs\") pod \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\" (UID: \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\") "
Nov 02 02:10:51.947630 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.947433    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-tmp-dir\") pod \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\" (UID: \"5f4b634d-f8b6-4faf-911b-12b3b7990fe1\") "
Nov 02 02:10:51.947781 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.947752    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-model-cache" pod "5f4b634d-f8b6-4faf-911b-12b3b7990fe1" (UID: "5f4b634d-f8b6-4faf-911b-12b3b7990fe1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:51.947972 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.947950    5639 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dd080e9-d0c8-4341-ac96-562165899a80-tls-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:51.948058 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.947977    5639 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-model-cache\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:51.948058 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.947976    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-home" pod "5f4b634d-f8b6-4faf-911b-12b3b7990fe1" (UID: "5f4b634d-f8b6-4faf-911b-12b3b7990fe1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:51.948058 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.947990    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-tmp-dir" pod "5f4b634d-f8b6-4faf-911b-12b3b7990fe1" (UID: "5f4b634d-f8b6-4faf-911b-12b3b7990fe1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:51.948058 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.947997    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c7r7\" (UniqueName: \"kubernetes.io/projected/0dd080e9-d0c8-4341-ac96-562165899a80-kube-api-access-4c7r7\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:51.948058 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.948032    5639 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-model-cache\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:51.948058 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.948049    5639 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dd080e9-d0c8-4341-ac96-562165899a80-dshm\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:51.950072 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.950051    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-dshm" pod "5f4b634d-f8b6-4faf-911b-12b3b7990fe1" (UID: "5f4b634d-f8b6-4faf-911b-12b3b7990fe1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:51.950325 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.950300    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-kube-api-access-bxftz" pod "5f4b634d-f8b6-4faf-911b-12b3b7990fe1" (UID: "5f4b634d-f8b6-4faf-911b-12b3b7990fe1"). InnerVolumeSpecName "kube-api-access-bxftz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:10:51.950565 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:51.950545    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-tls-certs" pod "5f4b634d-f8b6-4faf-911b-12b3b7990fe1" (UID: "5f4b634d-f8b6-4faf-911b-12b3b7990fe1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:10:52.049687 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:52.049612    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxftz\" (UniqueName: \"kubernetes.io/projected/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-kube-api-access-bxftz\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:52.049687 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:52.049641    5639 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-dshm\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:52.049687 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:52.049650    5639 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-home\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:52.049687 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:52.049658    5639 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-tls-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:52.049687 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:52.049666    5639 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f4b634d-f8b6-4faf-911b-12b3b7990fe1-tmp-dir\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:52.222129 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:52.222096    5639 generic.go:356] "Generic (PLEG): container finished" podID="0dd080e9-d0c8-4341-ac96-562165899a80" containerID="1eae94d5c379dfa2de7082f7beaa829d193398bae412fa0a567a99b565b26efe" exitCode=0
Nov 02 02:10:52.222363 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:52.222135    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"
Nov 02 02:10:52.222363 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:52.222175    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" event={"ID":"0dd080e9-d0c8-4341-ac96-562165899a80","Type":"ContainerDied","Data":"1eae94d5c379dfa2de7082f7beaa829d193398bae412fa0a567a99b565b26efe"}
Nov 02 02:10:52.222363 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:52.222223    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h" event={"ID":"0dd080e9-d0c8-4341-ac96-562165899a80","Type":"ContainerDied","Data":"2991ade83cbf6040d3013e0c4dfb234017882f1c1c1a9f8c5768af853faa11ad"}
Nov 02 02:10:52.222363 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:52.222247    5639 scope.go:122] "RemoveContainer" containerID="1eae94d5c379dfa2de7082f7beaa829d193398bae412fa0a567a99b565b26efe"
Nov 02 02:10:52.223728 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:52.223708    5639 generic.go:356] "Generic (PLEG): container finished" podID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerID="13427df22a25f684b69773ba0e4d9c76bd848aa0d1f0e56dfbb3f3b7298e5d37" exitCode=0
Nov 02 02:10:52.223834 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:52.223754    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" event={"ID":"5f4b634d-f8b6-4faf-911b-12b3b7990fe1","Type":"ContainerDied","Data":"13427df22a25f684b69773ba0e4d9c76bd848aa0d1f0e56dfbb3f3b7298e5d37"}
Nov 02 02:10:52.223834 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:52.223789    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6" event={"ID":"5f4b634d-f8b6-4faf-911b-12b3b7990fe1","Type":"ContainerDied","Data":"25644ce7742a511d0fa5e0c0c27bc85e3d89e29aa0c199878c1e363a1065aa32"}
Nov 02 02:10:52.223834 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:52.223827    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"
Nov 02 02:10:52.234191 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:52.234170    5639 scope.go:122] "RemoveContainer" containerID="1eae94d5c379dfa2de7082f7beaa829d193398bae412fa0a567a99b565b26efe"
Nov 02 02:10:52.234558 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:10:52.234538    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eae94d5c379dfa2de7082f7beaa829d193398bae412fa0a567a99b565b26efe\": container with ID starting with 1eae94d5c379dfa2de7082f7beaa829d193398bae412fa0a567a99b565b26efe not found: ID does not exist" containerID="1eae94d5c379dfa2de7082f7beaa829d193398bae412fa0a567a99b565b26efe"
Nov 02 02:10:52.234634 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:52.234565    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eae94d5c379dfa2de7082f7beaa829d193398bae412fa0a567a99b565b26efe"} err="failed to get container status \"1eae94d5c379dfa2de7082f7beaa829d193398bae412fa0a567a99b565b26efe\": rpc error: code = NotFound desc = could not find container \"1eae94d5c379dfa2de7082f7beaa829d193398bae412fa0a567a99b565b26efe\": container with ID starting with 1eae94d5c379dfa2de7082f7beaa829d193398bae412fa0a567a99b565b26efe not found: ID does not exist"
Nov 02 02:10:52.234634 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:52.234582    5639 scope.go:122] "RemoveContainer" containerID="13427df22a25f684b69773ba0e4d9c76bd848aa0d1f0e56dfbb3f3b7298e5d37"
Nov 02 02:10:52.243797 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:52.243779    5639 scope.go:122] "RemoveContainer" containerID="13427df22a25f684b69773ba0e4d9c76bd848aa0d1f0e56dfbb3f3b7298e5d37"
Nov 02 02:10:52.244042 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:10:52.244022    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13427df22a25f684b69773ba0e4d9c76bd848aa0d1f0e56dfbb3f3b7298e5d37\": container with ID starting with 13427df22a25f684b69773ba0e4d9c76bd848aa0d1f0e56dfbb3f3b7298e5d37 not found: ID does not exist" containerID="13427df22a25f684b69773ba0e4d9c76bd848aa0d1f0e56dfbb3f3b7298e5d37"
Nov 02 02:10:52.244110 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:52.244051    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13427df22a25f684b69773ba0e4d9c76bd848aa0d1f0e56dfbb3f3b7298e5d37"} err="failed to get container status \"13427df22a25f684b69773ba0e4d9c76bd848aa0d1f0e56dfbb3f3b7298e5d37\": rpc error: code = NotFound desc = could not find container \"13427df22a25f684b69773ba0e4d9c76bd848aa0d1f0e56dfbb3f3b7298e5d37\": container with ID starting with 13427df22a25f684b69773ba0e4d9c76bd848aa0d1f0e56dfbb3f3b7298e5d37 not found: ID does not exist"
Nov 02 02:10:52.387371 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:52.387334    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"]
Nov 02 02:10:52.391599 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:52.391574    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5888dc/simulator-w6-5888dc-kserve-749df46745-rvzn6"]
Nov 02 02:10:52.500698 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:52.500668    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"]
Nov 02 02:10:52.505239 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:52.505215    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5888dc/premium-w6-5888dc-kserve-69f8b55c64-j7s6h"]
Nov 02 02:10:53.217932 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:53.217896    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" path="/var/lib/kubelet/pods/0dd080e9-d0c8-4341-ac96-562165899a80/volumes"
Nov 02 02:10:53.218380 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:10:53.218365    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" path="/var/lib/kubelet/pods/5f4b634d-f8b6-4faf-911b-12b3b7990fe1/volumes"
Nov 02 02:10:57.427427 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:10:57.427387    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:11:00.384038 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:00.384002    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9039bf-pg48g"]
Nov 02 02:11:00.433486 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:00.433451    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9039bf-pg48g"]
Nov 02 02:11:01.219953 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:01.219914    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4aac84c-d970-4ec4-bcea-6c24f59a67bd" path="/var/lib/kubelet/pods/a4aac84c-d970-4ec4-bcea-6c24f59a67bd/volumes"
Nov 02 02:11:08.222663 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.222627    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd"]
Nov 02 02:11:08.223488 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.223468    5639 cpu_manager.go:414] "containerMap: removing container" podUID="add349ba-4927-476d-ab88-5336942c592f" containerName="revoke-keys"
Nov 02 02:11:08.223561 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.223491    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add349ba-4927-476d-ab88-5336942c592f" containerName="revoke-keys"
Nov 02 02:11:08.223561 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.223523    5639 cpu_manager.go:414] "containerMap: removing container" podUID="7b64c8f8-821f-4d7b-a684-40433fb5396c" containerName="istio-proxy"
Nov 02 02:11:08.223561 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.223530    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b64c8f8-821f-4d7b-a684-40433fb5396c" containerName="istio-proxy"
Nov 02 02:11:08.223561 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.223553    5639 cpu_manager.go:414] "containerMap: removing container" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main"
Nov 02 02:11:08.223561 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.223560    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main"
Nov 02 02:11:08.223714 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.223596    5639 cpu_manager.go:414] "containerMap: removing container" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main"
Nov 02 02:11:08.223714 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.223602    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main"
Nov 02 02:11:08.223714 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.223694    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd080e9-d0c8-4341-ac96-562165899a80" containerName="main"
Nov 02 02:11:08.223714 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.223710    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="add349ba-4927-476d-ab88-5336942c592f" containerName="revoke-keys"
Nov 02 02:11:08.223830 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.223717    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b64c8f8-821f-4d7b-a684-40433fb5396c" containerName="istio-proxy"
Nov 02 02:11:08.223830 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.223724    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f4b634d-f8b6-4faf-911b-12b3b7990fe1" containerName="main"
Nov 02 02:11:08.227138 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.227122    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd"
Nov 02 02:11:08.280164 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.280123    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:11:08.330136 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.330104    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:11:08.330136 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.330127    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:11:08.330433 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.330312    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:11:08.332669 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.332646    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd"]
Nov 02 02:11:08.340920 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.340894    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7b4f\" (UniqueName: \"kubernetes.io/projected/d8f6bc15-77ce-4b46-9a01-106f2518402a-kube-api-access-h7b4f\") pod \"maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd\" (UID: \"d8f6bc15-77ce-4b46-9a01-106f2518402a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd"
Nov 02 02:11:08.341064 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.340941    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f6bc15-77ce-4b46-9a01-106f2518402a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd\" (UID: \"d8f6bc15-77ce-4b46-9a01-106f2518402a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd"
Nov 02 02:11:08.341140 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.341109    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f6bc15-77ce-4b46-9a01-106f2518402a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd\" (UID: \"d8f6bc15-77ce-4b46-9a01-106f2518402a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd"
Nov 02 02:11:08.442859 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.442820    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7b4f\" (UniqueName: \"kubernetes.io/projected/d8f6bc15-77ce-4b46-9a01-106f2518402a-kube-api-access-h7b4f\") pod \"maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd\" (UID: \"d8f6bc15-77ce-4b46-9a01-106f2518402a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd"
Nov 02 02:11:08.442859 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.442868    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f6bc15-77ce-4b46-9a01-106f2518402a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd\" (UID: \"d8f6bc15-77ce-4b46-9a01-106f2518402a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd"
Nov 02 02:11:08.443083 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.442921    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f6bc15-77ce-4b46-9a01-106f2518402a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd\" (UID: \"d8f6bc15-77ce-4b46-9a01-106f2518402a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd"
Nov 02 02:11:08.443711 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.443690    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f6bc15-77ce-4b46-9a01-106f2518402a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd\" (UID: \"d8f6bc15-77ce-4b46-9a01-106f2518402a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd"
Nov 02 02:11:08.445445 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.445426    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f6bc15-77ce-4b46-9a01-106f2518402a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd\" (UID: \"d8f6bc15-77ce-4b46-9a01-106f2518402a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd"
Nov 02 02:11:08.503785 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.503713    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7b4f\" (UniqueName: \"kubernetes.io/projected/d8f6bc15-77ce-4b46-9a01-106f2518402a-kube-api-access-h7b4f\") pod \"maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd\" (UID: \"d8f6bc15-77ce-4b46-9a01-106f2518402a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd"
Nov 02 02:11:08.540174 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.539875    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd"
Nov 02 02:11:08.727878 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.727842    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:11:08.920032 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:08.919999    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd"]
Nov 02 02:11:08.922309 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:11:08.922252    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f6bc15_77ce_4b46_9a01_106f2518402a.slice/crio-b430813ee75d5d1d98bd3511bb8a9ed0955fa1bbfaa9c9602416c087f2459a31 WatchSource:0}: Error finding container b430813ee75d5d1d98bd3511bb8a9ed0955fa1bbfaa9c9602416c087f2459a31: Status 404 returned error can't find the container with id b430813ee75d5d1d98bd3511bb8a9ed0955fa1bbfaa9c9602416c087f2459a31
Nov 02 02:11:09.349801 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.349766    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd" event={"ID":"d8f6bc15-77ce-4b46-9a01-106f2518402a","Type":"ContainerStarted","Data":"0f1f770d8206e8c9de8ed4440152f237aed9ef54b24ccfe6b00e60d16c8434bf"}
Nov 02 02:11:09.350226 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.349825    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd" event={"ID":"d8f6bc15-77ce-4b46-9a01-106f2518402a","Type":"ContainerStarted","Data":"b430813ee75d5d1d98bd3511bb8a9ed0955fa1bbfaa9c9602416c087f2459a31"}
Nov 02 02:11:09.467258 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.467150    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd" podStartSLOduration=1.467135146 podStartE2EDuration="1.467135146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:11:09.464676137 +0000 UTC m=+2734.368358268" watchObservedRunningTime="2026-11-02 02:11:09.467135146 +0000 UTC m=+2734.370817258"
Nov 02 02:11:09.496827 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.496793    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdb6931-zhrmm"]
Nov 02 02:11:09.501049 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.501030    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdb6931-zhrmm"
Nov 02 02:11:09.558185 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.558147    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c70ecb0c-g82hr"]
Nov 02 02:11:09.562320 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.562291    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c70ecb0c-g82hr"
Nov 02 02:11:09.578965 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.578940    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdb6931-zhrmm"]
Nov 02 02:11:09.620382 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.620348    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c70ecb0c-g82hr"]
Nov 02 02:11:09.659448 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.659401    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkgfl\" (UniqueName: \"kubernetes.io/projected/78dba175-b3da-461b-b642-83a8fc049510-kube-api-access-fkgfl\") pod \"maas-api-revoke-sub-c70ecb0c-g82hr\" (UID: \"78dba175-b3da-461b-b642-83a8fc049510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c70ecb0c-g82hr"
Nov 02 02:11:09.659661 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.659471    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78dba175-b3da-461b-b642-83a8fc049510-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c70ecb0c-g82hr\" (UID: \"78dba175-b3da-461b-b642-83a8fc049510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c70ecb0c-g82hr"
Nov 02 02:11:09.659661 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.659600    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c25qm\" (UniqueName: \"kubernetes.io/projected/e914afa8-6a65-486c-868e-f98369c26f6e-kube-api-access-c25qm\") pod \"maas-api-revoke-sub-4bdb6931-zhrmm\" (UID: \"e914afa8-6a65-486c-868e-f98369c26f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdb6931-zhrmm"
Nov 02 02:11:09.659800 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.659692    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e914afa8-6a65-486c-868e-f98369c26f6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bdb6931-zhrmm\" (UID: \"e914afa8-6a65-486c-868e-f98369c26f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdb6931-zhrmm"
Nov 02 02:11:09.659800 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.659745    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e914afa8-6a65-486c-868e-f98369c26f6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bdb6931-zhrmm\" (UID: \"e914afa8-6a65-486c-868e-f98369c26f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdb6931-zhrmm"
Nov 02 02:11:09.659800 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.659782    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78dba175-b3da-461b-b642-83a8fc049510-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c70ecb0c-g82hr\" (UID: \"78dba175-b3da-461b-b642-83a8fc049510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c70ecb0c-g82hr"
Nov 02 02:11:09.761296 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.761247    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c25qm\" (UniqueName: \"kubernetes.io/projected/e914afa8-6a65-486c-868e-f98369c26f6e-kube-api-access-c25qm\") pod \"maas-api-revoke-sub-4bdb6931-zhrmm\" (UID: \"e914afa8-6a65-486c-868e-f98369c26f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdb6931-zhrmm"
Nov 02 02:11:09.761479 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.761332    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e914afa8-6a65-486c-868e-f98369c26f6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bdb6931-zhrmm\" (UID: \"e914afa8-6a65-486c-868e-f98369c26f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdb6931-zhrmm"
Nov 02 02:11:09.761479 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.761364    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e914afa8-6a65-486c-868e-f98369c26f6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bdb6931-zhrmm\" (UID: \"e914afa8-6a65-486c-868e-f98369c26f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdb6931-zhrmm"
Nov 02 02:11:09.761479 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.761390    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78dba175-b3da-461b-b642-83a8fc049510-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c70ecb0c-g82hr\" (UID: \"78dba175-b3da-461b-b642-83a8fc049510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c70ecb0c-g82hr"
Nov 02 02:11:09.761654 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.761511    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkgfl\" (UniqueName: \"kubernetes.io/projected/78dba175-b3da-461b-b642-83a8fc049510-kube-api-access-fkgfl\") pod \"maas-api-revoke-sub-c70ecb0c-g82hr\" (UID: \"78dba175-b3da-461b-b642-83a8fc049510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c70ecb0c-g82hr"
Nov 02 02:11:09.761654 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.761551    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78dba175-b3da-461b-b642-83a8fc049510-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c70ecb0c-g82hr\" (UID: \"78dba175-b3da-461b-b642-83a8fc049510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c70ecb0c-g82hr"
Nov 02 02:11:09.762209 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.762183    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e914afa8-6a65-486c-868e-f98369c26f6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bdb6931-zhrmm\" (UID: \"e914afa8-6a65-486c-868e-f98369c26f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdb6931-zhrmm"
Nov 02 02:11:09.762333 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.762245    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78dba175-b3da-461b-b642-83a8fc049510-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c70ecb0c-g82hr\" (UID: \"78dba175-b3da-461b-b642-83a8fc049510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c70ecb0c-g82hr"
Nov 02 02:11:09.763897 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.763869    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e914afa8-6a65-486c-868e-f98369c26f6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bdb6931-zhrmm\" (UID: \"e914afa8-6a65-486c-868e-f98369c26f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdb6931-zhrmm"
Nov 02 02:11:09.764059 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.764039    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78dba175-b3da-461b-b642-83a8fc049510-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c70ecb0c-g82hr\" (UID: \"78dba175-b3da-461b-b642-83a8fc049510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c70ecb0c-g82hr"
Nov 02 02:11:09.822386 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.822358    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c25qm\" (UniqueName: \"kubernetes.io/projected/e914afa8-6a65-486c-868e-f98369c26f6e-kube-api-access-c25qm\") pod \"maas-api-revoke-sub-4bdb6931-zhrmm\" (UID: \"e914afa8-6a65-486c-868e-f98369c26f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdb6931-zhrmm"
Nov 02 02:11:09.822552 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.822464    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkgfl\" (UniqueName: \"kubernetes.io/projected/78dba175-b3da-461b-b642-83a8fc049510-kube-api-access-fkgfl\") pod \"maas-api-revoke-sub-c70ecb0c-g82hr\" (UID: \"78dba175-b3da-461b-b642-83a8fc049510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c70ecb0c-g82hr"
Nov 02 02:11:09.876162 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:09.876123    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c70ecb0c-g82hr"
Nov 02 02:11:10.115220 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:10.115188    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdb6931-zhrmm"
Nov 02 02:11:10.253612 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:10.253539    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c70ecb0c-g82hr"]
Nov 02 02:11:10.256273 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:11:10.256249    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78dba175_b3da_461b_b642_83a8fc049510.slice/crio-ec335493746cb4b0f162628557e2301fc39876343695e593bc74d4db2c435cc0 WatchSource:0}: Error finding container ec335493746cb4b0f162628557e2301fc39876343695e593bc74d4db2c435cc0: Status 404 returned error can't find the container with id ec335493746cb4b0f162628557e2301fc39876343695e593bc74d4db2c435cc0
Nov 02 02:11:10.365522 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:10.365383    5639 generic.go:356] "Generic (PLEG): container finished" podID="d8f6bc15-77ce-4b46-9a01-106f2518402a" containerID="0f1f770d8206e8c9de8ed4440152f237aed9ef54b24ccfe6b00e60d16c8434bf" exitCode=0
Nov 02 02:11:10.366040 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:10.365567    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd" event={"ID":"d8f6bc15-77ce-4b46-9a01-106f2518402a","Type":"ContainerDied","Data":"0f1f770d8206e8c9de8ed4440152f237aed9ef54b24ccfe6b00e60d16c8434bf"}
Nov 02 02:11:10.367865 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:10.367824    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c70ecb0c-g82hr" event={"ID":"78dba175-b3da-461b-b642-83a8fc049510","Type":"ContainerStarted","Data":"ec335493746cb4b0f162628557e2301fc39876343695e593bc74d4db2c435cc0"}
Nov 02 02:11:10.518172 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:10.518135    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdb6931-zhrmm"]
Nov 02 02:11:10.520578 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:11:10.520547    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode914afa8_6a65_486c_868e_f98369c26f6e.slice/crio-f47c463998e177d7c7f4b7b9bd136e869394a7512a35f44304750a09420cee72 WatchSource:0}: Error finding container f47c463998e177d7c7f4b7b9bd136e869394a7512a35f44304750a09420cee72: Status 404 returned error can't find the container with id f47c463998e177d7c7f4b7b9bd136e869394a7512a35f44304750a09420cee72
Nov 02 02:11:11.376475 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:11.376447    5639 generic.go:356] "Generic (PLEG): container finished" podID="78dba175-b3da-461b-b642-83a8fc049510" containerID="237ff5de638a6d6f327f60a1e7f7f6f43a284f5ce70dbe286a8564e3146f77d6" exitCode=0
Nov 02 02:11:11.376808 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:11.376532    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c70ecb0c-g82hr" event={"ID":"78dba175-b3da-461b-b642-83a8fc049510","Type":"ContainerDied","Data":"237ff5de638a6d6f327f60a1e7f7f6f43a284f5ce70dbe286a8564e3146f77d6"}
Nov 02 02:11:11.378110 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:11.378088    5639 generic.go:356] "Generic (PLEG): container finished" podID="e914afa8-6a65-486c-868e-f98369c26f6e" containerID="de30ae11b86ca31778506f6d1a4a87755237204eeb7e4c3c91254466911cff91" exitCode=0
Nov 02 02:11:11.378247 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:11.378169    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdb6931-zhrmm" event={"ID":"e914afa8-6a65-486c-868e-f98369c26f6e","Type":"ContainerDied","Data":"de30ae11b86ca31778506f6d1a4a87755237204eeb7e4c3c91254466911cff91"}
Nov 02 02:11:11.378247 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:11.378218    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdb6931-zhrmm" event={"ID":"e914afa8-6a65-486c-868e-f98369c26f6e","Type":"ContainerStarted","Data":"f47c463998e177d7c7f4b7b9bd136e869394a7512a35f44304750a09420cee72"}
Nov 02 02:11:11.529410 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:11.529381    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd"
Nov 02 02:11:11.684817 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:11.684782    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8f6bc15-77ce-4b46-9a01-106f2518402a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f6bc15-77ce-4b46-9a01-106f2518402a-maas-api-openshift-ca\") pod \"d8f6bc15-77ce-4b46-9a01-106f2518402a\" (UID: \"d8f6bc15-77ce-4b46-9a01-106f2518402a\") "
Nov 02 02:11:11.684974 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:11.684911    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8f6bc15-77ce-4b46-9a01-106f2518402a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f6bc15-77ce-4b46-9a01-106f2518402a-maas-api-xks-ca\") pod \"d8f6bc15-77ce-4b46-9a01-106f2518402a\" (UID: \"d8f6bc15-77ce-4b46-9a01-106f2518402a\") "
Nov 02 02:11:11.685020 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:11.684972    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8f6bc15-77ce-4b46-9a01-106f2518402a-kube-api-access-h7b4f\" (UniqueName: \"kubernetes.io/projected/d8f6bc15-77ce-4b46-9a01-106f2518402a-kube-api-access-h7b4f\") pod \"d8f6bc15-77ce-4b46-9a01-106f2518402a\" (UID: \"d8f6bc15-77ce-4b46-9a01-106f2518402a\") "
Nov 02 02:11:11.685137 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:11.685114    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8f6bc15-77ce-4b46-9a01-106f2518402a-maas-api-openshift-ca" pod "d8f6bc15-77ce-4b46-9a01-106f2518402a" (UID: "d8f6bc15-77ce-4b46-9a01-106f2518402a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:11:11.685410 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:11.685392    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f6bc15-77ce-4b46-9a01-106f2518402a-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:11.687304 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:11.687262    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8f6bc15-77ce-4b46-9a01-106f2518402a-maas-api-xks-ca" pod "d8f6bc15-77ce-4b46-9a01-106f2518402a" (UID: "d8f6bc15-77ce-4b46-9a01-106f2518402a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:11:11.687397 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:11.687266    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f6bc15-77ce-4b46-9a01-106f2518402a-kube-api-access-h7b4f" pod "d8f6bc15-77ce-4b46-9a01-106f2518402a" (UID: "d8f6bc15-77ce-4b46-9a01-106f2518402a"). InnerVolumeSpecName "kube-api-access-h7b4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:11.786949 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:11.786861    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f6bc15-77ce-4b46-9a01-106f2518402a-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:11.786949 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:11.786893    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7b4f\" (UniqueName: \"kubernetes.io/projected/d8f6bc15-77ce-4b46-9a01-106f2518402a-kube-api-access-h7b4f\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:12.386411 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:12.386355    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd" event={"ID":"d8f6bc15-77ce-4b46-9a01-106f2518402a","Type":"ContainerDied","Data":"b430813ee75d5d1d98bd3511bb8a9ed0955fa1bbfaa9c9602416c087f2459a31"}
Nov 02 02:11:12.386901 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:12.386431    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b430813ee75d5d1d98bd3511bb8a9ed0955fa1bbfaa9c9602416c087f2459a31"
Nov 02 02:11:12.386901 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:12.386376    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd"
Nov 02 02:11:12.555405 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:12.555377    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c70ecb0c-g82hr"
Nov 02 02:11:12.577820 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:12.577795    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdb6931-zhrmm"
Nov 02 02:11:12.606471 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:12.606434    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e914afa8-6a65-486c-868e-f98369c26f6e-kube-api-access-c25qm\" (UniqueName: \"kubernetes.io/projected/e914afa8-6a65-486c-868e-f98369c26f6e-kube-api-access-c25qm\") pod \"e914afa8-6a65-486c-868e-f98369c26f6e\" (UID: \"e914afa8-6a65-486c-868e-f98369c26f6e\") "
Nov 02 02:11:12.606620 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:12.606535    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e914afa8-6a65-486c-868e-f98369c26f6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e914afa8-6a65-486c-868e-f98369c26f6e-maas-api-openshift-ca\") pod \"e914afa8-6a65-486c-868e-f98369c26f6e\" (UID: \"e914afa8-6a65-486c-868e-f98369c26f6e\") "
Nov 02 02:11:12.606620 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:12.606563    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78dba175-b3da-461b-b642-83a8fc049510-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78dba175-b3da-461b-b642-83a8fc049510-maas-api-xks-ca\") pod \"78dba175-b3da-461b-b642-83a8fc049510\" (UID: \"78dba175-b3da-461b-b642-83a8fc049510\") "
Nov 02 02:11:12.606620 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:12.606603    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78dba175-b3da-461b-b642-83a8fc049510-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78dba175-b3da-461b-b642-83a8fc049510-maas-api-openshift-ca\") pod \"78dba175-b3da-461b-b642-83a8fc049510\" (UID: \"78dba175-b3da-461b-b642-83a8fc049510\") "
Nov 02 02:11:12.606828 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:12.606643    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e914afa8-6a65-486c-868e-f98369c26f6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e914afa8-6a65-486c-868e-f98369c26f6e-maas-api-xks-ca\") pod \"e914afa8-6a65-486c-868e-f98369c26f6e\" (UID: \"e914afa8-6a65-486c-868e-f98369c26f6e\") "
Nov 02 02:11:12.606828 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:12.606701    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78dba175-b3da-461b-b642-83a8fc049510-kube-api-access-fkgfl\" (UniqueName: \"kubernetes.io/projected/78dba175-b3da-461b-b642-83a8fc049510-kube-api-access-fkgfl\") pod \"78dba175-b3da-461b-b642-83a8fc049510\" (UID: \"78dba175-b3da-461b-b642-83a8fc049510\") "
Nov 02 02:11:12.606946 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:12.606920    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e914afa8-6a65-486c-868e-f98369c26f6e-maas-api-openshift-ca" pod "e914afa8-6a65-486c-868e-f98369c26f6e" (UID: "e914afa8-6a65-486c-868e-f98369c26f6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:11:12.607024 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:12.606990    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78dba175-b3da-461b-b642-83a8fc049510-maas-api-openshift-ca" pod "78dba175-b3da-461b-b642-83a8fc049510" (UID: "78dba175-b3da-461b-b642-83a8fc049510"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:11:12.607290 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:12.607241    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e914afa8-6a65-486c-868e-f98369c26f6e-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:12.607416 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:12.607270    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78dba175-b3da-461b-b642-83a8fc049510-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:12.609062 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:12.609036    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e914afa8-6a65-486c-868e-f98369c26f6e-maas-api-xks-ca" pod "e914afa8-6a65-486c-868e-f98369c26f6e" (UID: "e914afa8-6a65-486c-868e-f98369c26f6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:11:12.609200 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:12.609176    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78dba175-b3da-461b-b642-83a8fc049510-kube-api-access-fkgfl" pod "78dba175-b3da-461b-b642-83a8fc049510" (UID: "78dba175-b3da-461b-b642-83a8fc049510"). InnerVolumeSpecName "kube-api-access-fkgfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:12.609344 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:12.609252    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78dba175-b3da-461b-b642-83a8fc049510-maas-api-xks-ca" pod "78dba175-b3da-461b-b642-83a8fc049510" (UID: "78dba175-b3da-461b-b642-83a8fc049510"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:11:12.609344 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:12.609268    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e914afa8-6a65-486c-868e-f98369c26f6e-kube-api-access-c25qm" pod "e914afa8-6a65-486c-868e-f98369c26f6e" (UID: "e914afa8-6a65-486c-868e-f98369c26f6e"). InnerVolumeSpecName "kube-api-access-c25qm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:12.708140 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:12.708069    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c25qm\" (UniqueName: \"kubernetes.io/projected/e914afa8-6a65-486c-868e-f98369c26f6e-kube-api-access-c25qm\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:12.708140 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:12.708095    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78dba175-b3da-461b-b642-83a8fc049510-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:12.708140 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:12.708104    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e914afa8-6a65-486c-868e-f98369c26f6e-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:12.708140 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:12.708114    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkgfl\" (UniqueName: \"kubernetes.io/projected/78dba175-b3da-461b-b642-83a8fc049510-kube-api-access-fkgfl\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:12.724370 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:12.724336    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"]
Nov 02 02:11:12.734352 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:12.734315    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"]
Nov 02 02:11:13.393567 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:13.393530    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdb6931-zhrmm"
Nov 02 02:11:13.393567 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:13.393553    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdb6931-zhrmm" event={"ID":"e914afa8-6a65-486c-868e-f98369c26f6e","Type":"ContainerDied","Data":"f47c463998e177d7c7f4b7b9bd136e869394a7512a35f44304750a09420cee72"}
Nov 02 02:11:13.394051 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:13.393614    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f47c463998e177d7c7f4b7b9bd136e869394a7512a35f44304750a09420cee72"
Nov 02 02:11:13.395028 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:13.394994    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c70ecb0c-g82hr" event={"ID":"78dba175-b3da-461b-b642-83a8fc049510","Type":"ContainerDied","Data":"ec335493746cb4b0f162628557e2301fc39876343695e593bc74d4db2c435cc0"}
Nov 02 02:11:13.395166 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:13.395051    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec335493746cb4b0f162628557e2301fc39876343695e593bc74d4db2c435cc0"
Nov 02 02:11:13.395166 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:13.395011    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c70ecb0c-g82hr"
Nov 02 02:11:19.685867 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:19.685831    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24"]
Nov 02 02:11:19.794054 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:19.794016    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6"]
Nov 02 02:11:24.308188 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.308151    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee06273b-g8csb"]
Nov 02 02:11:24.309004 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.308983    5639 cpu_manager.go:414] "containerMap: removing container" podUID="d8f6bc15-77ce-4b46-9a01-106f2518402a" containerName="revoke-keys"
Nov 02 02:11:24.309054 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.309009    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8f6bc15-77ce-4b46-9a01-106f2518402a" containerName="revoke-keys"
Nov 02 02:11:24.309054 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.309046    5639 cpu_manager.go:414] "containerMap: removing container" podUID="78dba175-b3da-461b-b642-83a8fc049510" containerName="revoke-keys"
Nov 02 02:11:24.309124 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.309053    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78dba175-b3da-461b-b642-83a8fc049510" containerName="revoke-keys"
Nov 02 02:11:24.309124 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.309097    5639 cpu_manager.go:414] "containerMap: removing container" podUID="e914afa8-6a65-486c-868e-f98369c26f6e" containerName="revoke-keys"
Nov 02 02:11:24.309124 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.309107    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e914afa8-6a65-486c-868e-f98369c26f6e" containerName="revoke-keys"
Nov 02 02:11:24.309228 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.309201    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="e914afa8-6a65-486c-868e-f98369c26f6e" containerName="revoke-keys"
Nov 02 02:11:24.309228 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.309215    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8f6bc15-77ce-4b46-9a01-106f2518402a" containerName="revoke-keys"
Nov 02 02:11:24.309228 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.309225    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="78dba175-b3da-461b-b642-83a8fc049510" containerName="revoke-keys"
Nov 02 02:11:24.312704 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.312685    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee06273b-g8csb"
Nov 02 02:11:24.366324 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.366288    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38127a13-fkrm6"]
Nov 02 02:11:24.370633 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.370607    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38127a13-fkrm6"
Nov 02 02:11:24.371430 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.371405    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee06273b-g8csb"]
Nov 02 02:11:24.413672 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.413638    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:11:24.413672 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.413637    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:11:24.417646 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.417624    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:11:24.417646 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.417629    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:11:24.427249 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.427222    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38127a13-fkrm6"]
Nov 02 02:11:24.448947 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.448914    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bccf3fd7-7bdd-4e45-9781-6c64d977c616-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee06273b-g8csb\" (UID: \"bccf3fd7-7bdd-4e45-9781-6c64d977c616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee06273b-g8csb"
Nov 02 02:11:24.449156 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.449048    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t2dz\" (UniqueName: \"kubernetes.io/projected/826a3aa6-79b4-4397-9f69-8563b45453e2-kube-api-access-5t2dz\") pod \"maas-api-revoke-sub-38127a13-fkrm6\" (UID: \"826a3aa6-79b4-4397-9f69-8563b45453e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38127a13-fkrm6"
Nov 02 02:11:24.449156 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.449115    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bccf3fd7-7bdd-4e45-9781-6c64d977c616-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee06273b-g8csb\" (UID: \"bccf3fd7-7bdd-4e45-9781-6c64d977c616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee06273b-g8csb"
Nov 02 02:11:24.449156 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.449139    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/826a3aa6-79b4-4397-9f69-8563b45453e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38127a13-fkrm6\" (UID: \"826a3aa6-79b4-4397-9f69-8563b45453e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38127a13-fkrm6"
Nov 02 02:11:24.449324 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.449244    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qls27\" (UniqueName: \"kubernetes.io/projected/bccf3fd7-7bdd-4e45-9781-6c64d977c616-kube-api-access-qls27\") pod \"maas-api-revoke-sub-ee06273b-g8csb\" (UID: \"bccf3fd7-7bdd-4e45-9781-6c64d977c616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee06273b-g8csb"
Nov 02 02:11:24.449324 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.449319    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/826a3aa6-79b4-4397-9f69-8563b45453e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38127a13-fkrm6\" (UID: \"826a3aa6-79b4-4397-9f69-8563b45453e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38127a13-fkrm6"
Nov 02 02:11:24.550154 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.550120    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bccf3fd7-7bdd-4e45-9781-6c64d977c616-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee06273b-g8csb\" (UID: \"bccf3fd7-7bdd-4e45-9781-6c64d977c616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee06273b-g8csb"
Nov 02 02:11:24.550154 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.550154    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/826a3aa6-79b4-4397-9f69-8563b45453e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38127a13-fkrm6\" (UID: \"826a3aa6-79b4-4397-9f69-8563b45453e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38127a13-fkrm6"
Nov 02 02:11:24.550480 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.550206    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qls27\" (UniqueName: \"kubernetes.io/projected/bccf3fd7-7bdd-4e45-9781-6c64d977c616-kube-api-access-qls27\") pod \"maas-api-revoke-sub-ee06273b-g8csb\" (UID: \"bccf3fd7-7bdd-4e45-9781-6c64d977c616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee06273b-g8csb"
Nov 02 02:11:24.550480 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.550250    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/826a3aa6-79b4-4397-9f69-8563b45453e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38127a13-fkrm6\" (UID: \"826a3aa6-79b4-4397-9f69-8563b45453e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38127a13-fkrm6"
Nov 02 02:11:24.550480 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.550330    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bccf3fd7-7bdd-4e45-9781-6c64d977c616-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee06273b-g8csb\" (UID: \"bccf3fd7-7bdd-4e45-9781-6c64d977c616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee06273b-g8csb"
Nov 02 02:11:24.550480 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.550406    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t2dz\" (UniqueName: \"kubernetes.io/projected/826a3aa6-79b4-4397-9f69-8563b45453e2-kube-api-access-5t2dz\") pod \"maas-api-revoke-sub-38127a13-fkrm6\" (UID: \"826a3aa6-79b4-4397-9f69-8563b45453e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38127a13-fkrm6"
Nov 02 02:11:24.550898 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.550865    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bccf3fd7-7bdd-4e45-9781-6c64d977c616-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee06273b-g8csb\" (UID: \"bccf3fd7-7bdd-4e45-9781-6c64d977c616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee06273b-g8csb"
Nov 02 02:11:24.551028 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.550963    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/826a3aa6-79b4-4397-9f69-8563b45453e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38127a13-fkrm6\" (UID: \"826a3aa6-79b4-4397-9f69-8563b45453e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38127a13-fkrm6"
Nov 02 02:11:24.552864 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.552836    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/826a3aa6-79b4-4397-9f69-8563b45453e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38127a13-fkrm6\" (UID: \"826a3aa6-79b4-4397-9f69-8563b45453e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38127a13-fkrm6"
Nov 02 02:11:24.552957 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.552879    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bccf3fd7-7bdd-4e45-9781-6c64d977c616-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee06273b-g8csb\" (UID: \"bccf3fd7-7bdd-4e45-9781-6c64d977c616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee06273b-g8csb"
Nov 02 02:11:24.611694 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.611660    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t2dz\" (UniqueName: \"kubernetes.io/projected/826a3aa6-79b4-4397-9f69-8563b45453e2-kube-api-access-5t2dz\") pod \"maas-api-revoke-sub-38127a13-fkrm6\" (UID: \"826a3aa6-79b4-4397-9f69-8563b45453e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38127a13-fkrm6"
Nov 02 02:11:24.611909 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.611893    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qls27\" (UniqueName: \"kubernetes.io/projected/bccf3fd7-7bdd-4e45-9781-6c64d977c616-kube-api-access-qls27\") pod \"maas-api-revoke-sub-ee06273b-g8csb\" (UID: \"bccf3fd7-7bdd-4e45-9781-6c64d977c616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee06273b-g8csb"
Nov 02 02:11:24.626932 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.626899    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee06273b-g8csb"
Nov 02 02:11:24.685852 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.685821    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38127a13-fkrm6"
Nov 02 02:11:24.686168 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.686114    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24" podUID="5b302fb0-288a-4d1d-aece-d87bdf645f92" containerName="payload-processing" containerID="cri-o://da409789a210797ce7fe0a508631a8a02e6494af1eaeae1c2793c48e65fd22e1" gracePeriod=25
Nov 02 02:11:24.794823 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.794748    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6" podUID="b145a144-84c0-43c6-b08b-c5c196efddf9" containerName="payload-pre-processing" containerID="cri-o://9da497a8309393231941ea3b21c7741a1c7a7bbaa0068ccc5f7c208c970b892c" gracePeriod=25
Nov 02 02:11:24.938601 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:24.938575    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24"
Nov 02 02:11:25.009104 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.009067    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee06273b-g8csb"]
Nov 02 02:11:25.009832 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:11:25.009784    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbccf3fd7_7bdd_4e45_9781_6c64d977c616.slice/crio-1f066848ec8eeb8a68a3dc2731e054f2ca99f7c2edae5b8baca1abe184eb7daa WatchSource:0}: Error finding container 1f066848ec8eeb8a68a3dc2731e054f2ca99f7c2edae5b8baca1abe184eb7daa: Status 404 returned error can't find the container with id 1f066848ec8eeb8a68a3dc2731e054f2ca99f7c2edae5b8baca1abe184eb7daa
Nov 02 02:11:25.024257 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.023248    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01624761-nqfb5"]
Nov 02 02:11:25.024257 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.024091    5639 cpu_manager.go:414] "containerMap: removing container" podUID="5b302fb0-288a-4d1d-aece-d87bdf645f92" containerName="payload-processing"
Nov 02 02:11:25.024257 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.024110    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b302fb0-288a-4d1d-aece-d87bdf645f92" containerName="payload-processing"
Nov 02 02:11:25.024257 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.024251    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b302fb0-288a-4d1d-aece-d87bdf645f92" containerName="payload-processing"
Nov 02 02:11:25.030868 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.030844    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01624761-nqfb5"
Nov 02 02:11:25.056444 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.056420    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b302fb0-288a-4d1d-aece-d87bdf645f92-kube-api-access-q6p66\" (UniqueName: \"kubernetes.io/projected/5b302fb0-288a-4d1d-aece-d87bdf645f92-kube-api-access-q6p66\") pod \"5b302fb0-288a-4d1d-aece-d87bdf645f92\" (UID: \"5b302fb0-288a-4d1d-aece-d87bdf645f92\") "
Nov 02 02:11:25.056598 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.056539    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b302fb0-288a-4d1d-aece-d87bdf645f92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b302fb0-288a-4d1d-aece-d87bdf645f92-plugins-config-volume\") pod \"5b302fb0-288a-4d1d-aece-d87bdf645f92\" (UID: \"5b302fb0-288a-4d1d-aece-d87bdf645f92\") "
Nov 02 02:11:25.056967 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.056936    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b302fb0-288a-4d1d-aece-d87bdf645f92-plugins-config-volume" pod "5b302fb0-288a-4d1d-aece-d87bdf645f92" (UID: "5b302fb0-288a-4d1d-aece-d87bdf645f92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:11:25.059589 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.059563    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b302fb0-288a-4d1d-aece-d87bdf645f92-kube-api-access-q6p66" pod "5b302fb0-288a-4d1d-aece-d87bdf645f92" (UID: "5b302fb0-288a-4d1d-aece-d87bdf645f92"). InnerVolumeSpecName "kube-api-access-q6p66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:25.065910 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.065886    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6"
Nov 02 02:11:25.084705 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.084673    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01624761-nqfb5"]
Nov 02 02:11:25.118611 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.118563    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38127a13-fkrm6"]
Nov 02 02:11:25.120032 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:11:25.120005    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826a3aa6_79b4_4397_9f69_8563b45453e2.slice/crio-c4b627a8204fa64a95a65bd37946f5255598748a3d1fe4dfda562a2091df623c WatchSource:0}: Error finding container c4b627a8204fa64a95a65bd37946f5255598748a3d1fe4dfda562a2091df623c: Status 404 returned error can't find the container with id c4b627a8204fa64a95a65bd37946f5255598748a3d1fe4dfda562a2091df623c
Nov 02 02:11:25.157377 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.157355    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b145a144-84c0-43c6-b08b-c5c196efddf9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b145a144-84c0-43c6-b08b-c5c196efddf9-plugins-config-volume\") pod \"b145a144-84c0-43c6-b08b-c5c196efddf9\" (UID: \"b145a144-84c0-43c6-b08b-c5c196efddf9\") "
Nov 02 02:11:25.157485 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.157447    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b145a144-84c0-43c6-b08b-c5c196efddf9-kube-api-access-c7lkx\" (UniqueName: \"kubernetes.io/projected/b145a144-84c0-43c6-b08b-c5c196efddf9-kube-api-access-c7lkx\") pod \"b145a144-84c0-43c6-b08b-c5c196efddf9\" (UID: \"b145a144-84c0-43c6-b08b-c5c196efddf9\") "
Nov 02 02:11:25.157763 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.157740    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b145a144-84c0-43c6-b08b-c5c196efddf9-plugins-config-volume" pod "b145a144-84c0-43c6-b08b-c5c196efddf9" (UID: "b145a144-84c0-43c6-b08b-c5c196efddf9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:11:25.157825 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.157809    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d97a1388-6ca5-4b25-8a9d-d651faf46527-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01624761-nqfb5\" (UID: \"d97a1388-6ca5-4b25-8a9d-d651faf46527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01624761-nqfb5"
Nov 02 02:11:25.157885 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.157873    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lxwd\" (UniqueName: \"kubernetes.io/projected/d97a1388-6ca5-4b25-8a9d-d651faf46527-kube-api-access-6lxwd\") pod \"maas-api-revoke-sub-01624761-nqfb5\" (UID: \"d97a1388-6ca5-4b25-8a9d-d651faf46527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01624761-nqfb5"
Nov 02 02:11:25.158057 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.158034    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d97a1388-6ca5-4b25-8a9d-d651faf46527-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01624761-nqfb5\" (UID: \"d97a1388-6ca5-4b25-8a9d-d651faf46527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01624761-nqfb5"
Nov 02 02:11:25.158237 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.158224    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b302fb0-288a-4d1d-aece-d87bdf645f92-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:25.158299 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.158240    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b145a144-84c0-43c6-b08b-c5c196efddf9-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:25.158299 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.158252    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6p66\" (UniqueName: \"kubernetes.io/projected/5b302fb0-288a-4d1d-aece-d87bdf645f92-kube-api-access-q6p66\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:25.159581 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.159560    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b145a144-84c0-43c6-b08b-c5c196efddf9-kube-api-access-c7lkx" pod "b145a144-84c0-43c6-b08b-c5c196efddf9" (UID: "b145a144-84c0-43c6-b08b-c5c196efddf9"). InnerVolumeSpecName "kube-api-access-c7lkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:25.259639 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.259608    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d97a1388-6ca5-4b25-8a9d-d651faf46527-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01624761-nqfb5\" (UID: \"d97a1388-6ca5-4b25-8a9d-d651faf46527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01624761-nqfb5"
Nov 02 02:11:25.259788 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.259670    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lxwd\" (UniqueName: \"kubernetes.io/projected/d97a1388-6ca5-4b25-8a9d-d651faf46527-kube-api-access-6lxwd\") pod \"maas-api-revoke-sub-01624761-nqfb5\" (UID: \"d97a1388-6ca5-4b25-8a9d-d651faf46527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01624761-nqfb5"
Nov 02 02:11:25.259788 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.259724    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d97a1388-6ca5-4b25-8a9d-d651faf46527-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01624761-nqfb5\" (UID: \"d97a1388-6ca5-4b25-8a9d-d651faf46527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01624761-nqfb5"
Nov 02 02:11:25.259901 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.259788    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7lkx\" (UniqueName: \"kubernetes.io/projected/b145a144-84c0-43c6-b08b-c5c196efddf9-kube-api-access-c7lkx\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:25.260424 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.260396    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d97a1388-6ca5-4b25-8a9d-d651faf46527-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01624761-nqfb5\" (UID: \"d97a1388-6ca5-4b25-8a9d-d651faf46527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01624761-nqfb5"
Nov 02 02:11:25.262722 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.262694    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d97a1388-6ca5-4b25-8a9d-d651faf46527-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01624761-nqfb5\" (UID: \"d97a1388-6ca5-4b25-8a9d-d651faf46527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01624761-nqfb5"
Nov 02 02:11:25.320954 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.320919    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lxwd\" (UniqueName: \"kubernetes.io/projected/d97a1388-6ca5-4b25-8a9d-d651faf46527-kube-api-access-6lxwd\") pod \"maas-api-revoke-sub-01624761-nqfb5\" (UID: \"d97a1388-6ca5-4b25-8a9d-d651faf46527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01624761-nqfb5"
Nov 02 02:11:25.363923 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.363893    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01624761-nqfb5"
Nov 02 02:11:25.476133 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.476096    5639 generic.go:356] "Generic (PLEG): container finished" podID="5b302fb0-288a-4d1d-aece-d87bdf645f92" containerID="da409789a210797ce7fe0a508631a8a02e6494af1eaeae1c2793c48e65fd22e1" exitCode=0
Nov 02 02:11:25.476336 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.476158    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24"
Nov 02 02:11:25.476336 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.476171    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24" event={"ID":"5b302fb0-288a-4d1d-aece-d87bdf645f92","Type":"ContainerDied","Data":"da409789a210797ce7fe0a508631a8a02e6494af1eaeae1c2793c48e65fd22e1"}
Nov 02 02:11:25.476336 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.476224    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24" event={"ID":"5b302fb0-288a-4d1d-aece-d87bdf645f92","Type":"ContainerDied","Data":"91b9e9985a17c8f97681ff21aaa60a26af8ea0a6a591b130398ed2a9c2f0e6b1"}
Nov 02 02:11:25.476336 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.476255    5639 scope.go:122] "RemoveContainer" containerID="da409789a210797ce7fe0a508631a8a02e6494af1eaeae1c2793c48e65fd22e1"
Nov 02 02:11:25.478189 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.478160    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38127a13-fkrm6" event={"ID":"826a3aa6-79b4-4397-9f69-8563b45453e2","Type":"ContainerStarted","Data":"844ba007e78ce4ad32234aeae1adbb5b91c1f7914aa6ac48729a4bf873a2b2b7"}
Nov 02 02:11:25.478400 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.478220    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38127a13-fkrm6" event={"ID":"826a3aa6-79b4-4397-9f69-8563b45453e2","Type":"ContainerStarted","Data":"c4b627a8204fa64a95a65bd37946f5255598748a3d1fe4dfda562a2091df623c"}
Nov 02 02:11:25.480132 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.480108    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee06273b-g8csb" event={"ID":"bccf3fd7-7bdd-4e45-9781-6c64d977c616","Type":"ContainerStarted","Data":"1650be9706249db165f5905e674a23e38d4e56658fcdbbb86562b4b7917ee456"}
Nov 02 02:11:25.480247 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.480153    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee06273b-g8csb" event={"ID":"bccf3fd7-7bdd-4e45-9781-6c64d977c616","Type":"ContainerStarted","Data":"1f066848ec8eeb8a68a3dc2731e054f2ca99f7c2edae5b8baca1abe184eb7daa"}
Nov 02 02:11:25.481629 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.481607    5639 generic.go:356] "Generic (PLEG): container finished" podID="b145a144-84c0-43c6-b08b-c5c196efddf9" containerID="9da497a8309393231941ea3b21c7741a1c7a7bbaa0068ccc5f7c208c970b892c" exitCode=0
Nov 02 02:11:25.481737 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.481659    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6" event={"ID":"b145a144-84c0-43c6-b08b-c5c196efddf9","Type":"ContainerDied","Data":"9da497a8309393231941ea3b21c7741a1c7a7bbaa0068ccc5f7c208c970b892c"}
Nov 02 02:11:25.481737 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.481685    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6" event={"ID":"b145a144-84c0-43c6-b08b-c5c196efddf9","Type":"ContainerDied","Data":"c21d1fe6a85b954690919f129a0571f9e7b9449a8c6e9be2840385bb9fa0e242"}
Nov 02 02:11:25.481737 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.481660    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6"
Nov 02 02:11:25.487328 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.487265    5639 scope.go:122] "RemoveContainer" containerID="da409789a210797ce7fe0a508631a8a02e6494af1eaeae1c2793c48e65fd22e1"
Nov 02 02:11:25.487560 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:11:25.487541    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da409789a210797ce7fe0a508631a8a02e6494af1eaeae1c2793c48e65fd22e1\": container with ID starting with da409789a210797ce7fe0a508631a8a02e6494af1eaeae1c2793c48e65fd22e1 not found: ID does not exist" containerID="da409789a210797ce7fe0a508631a8a02e6494af1eaeae1c2793c48e65fd22e1"
Nov 02 02:11:25.487617 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.487569    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da409789a210797ce7fe0a508631a8a02e6494af1eaeae1c2793c48e65fd22e1"} err="failed to get container status \"da409789a210797ce7fe0a508631a8a02e6494af1eaeae1c2793c48e65fd22e1\": rpc error: code = NotFound desc = could not find container \"da409789a210797ce7fe0a508631a8a02e6494af1eaeae1c2793c48e65fd22e1\": container with ID starting with da409789a210797ce7fe0a508631a8a02e6494af1eaeae1c2793c48e65fd22e1 not found: ID does not exist"
Nov 02 02:11:25.487617 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.487585    5639 scope.go:122] "RemoveContainer" containerID="9da497a8309393231941ea3b21c7741a1c7a7bbaa0068ccc5f7c208c970b892c"
Nov 02 02:11:25.497054 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.497034    5639 scope.go:122] "RemoveContainer" containerID="9da497a8309393231941ea3b21c7741a1c7a7bbaa0068ccc5f7c208c970b892c"
Nov 02 02:11:25.497433 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:11:25.497410    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9da497a8309393231941ea3b21c7741a1c7a7bbaa0068ccc5f7c208c970b892c\": container with ID starting with 9da497a8309393231941ea3b21c7741a1c7a7bbaa0068ccc5f7c208c970b892c not found: ID does not exist" containerID="9da497a8309393231941ea3b21c7741a1c7a7bbaa0068ccc5f7c208c970b892c"
Nov 02 02:11:25.497501 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.497442    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9da497a8309393231941ea3b21c7741a1c7a7bbaa0068ccc5f7c208c970b892c"} err="failed to get container status \"9da497a8309393231941ea3b21c7741a1c7a7bbaa0068ccc5f7c208c970b892c\": rpc error: code = NotFound desc = could not find container \"9da497a8309393231941ea3b21c7741a1c7a7bbaa0068ccc5f7c208c970b892c\": container with ID starting with 9da497a8309393231941ea3b21c7741a1c7a7bbaa0068ccc5f7c208c970b892c not found: ID does not exist"
Nov 02 02:11:25.645193 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.645086    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee06273b-g8csb" podStartSLOduration=1.6450734439999999 podStartE2EDuration="1.645073444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:11:25.643830503 +0000 UTC m=+2750.547512691" watchObservedRunningTime="2026-11-02 02:11:25.645073444 +0000 UTC m=+2750.548755559"
Nov 02 02:11:25.739264 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.739229    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01624761-nqfb5"]
Nov 02 02:11:25.741583 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:11:25.741556    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd97a1388_6ca5_4b25_8a9d_d651faf46527.slice/crio-a6734fbd301afe565e147c27a65cbe2ea13e434963f72749f3ec7580115b47cf WatchSource:0}: Error finding container a6734fbd301afe565e147c27a65cbe2ea13e434963f72749f3ec7580115b47cf: Status 404 returned error can't find the container with id a6734fbd301afe565e147c27a65cbe2ea13e434963f72749f3ec7580115b47cf
Nov 02 02:11:25.756881 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.756859    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6"]
Nov 02 02:11:25.764645 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.764617    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-88e76d-65b4b49575-cz9l6"]
Nov 02 02:11:25.872762 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.872733    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24"]
Nov 02 02:11:25.876339 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.876313    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-88e76d-dc5bb4985-6ww24"]
Nov 02 02:11:25.990607 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:25.990558    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38127a13-fkrm6" podStartSLOduration=1.990543173 podStartE2EDuration="1.990543173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:11:25.988571726 +0000 UTC m=+2750.892253840" watchObservedRunningTime="2026-11-02 02:11:25.990543173 +0000 UTC m=+2750.894225287"
Nov 02 02:11:26.494799 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:26.494149    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01624761-nqfb5" event={"ID":"d97a1388-6ca5-4b25-8a9d-d651faf46527","Type":"ContainerStarted","Data":"a29520a468861a3629622900e57eea8da284fa66ad4cc0409ee70db163086a29"}
Nov 02 02:11:26.494799 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:26.494204    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01624761-nqfb5" event={"ID":"d97a1388-6ca5-4b25-8a9d-d651faf46527","Type":"ContainerStarted","Data":"a6734fbd301afe565e147c27a65cbe2ea13e434963f72749f3ec7580115b47cf"}
Nov 02 02:11:26.496312 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:26.496255    5639 generic.go:356] "Generic (PLEG): container finished" podID="826a3aa6-79b4-4397-9f69-8563b45453e2" containerID="844ba007e78ce4ad32234aeae1adbb5b91c1f7914aa6ac48729a4bf873a2b2b7" exitCode=0
Nov 02 02:11:26.496434 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:26.496317    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38127a13-fkrm6" event={"ID":"826a3aa6-79b4-4397-9f69-8563b45453e2","Type":"ContainerDied","Data":"844ba007e78ce4ad32234aeae1adbb5b91c1f7914aa6ac48729a4bf873a2b2b7"}
Nov 02 02:11:26.498406 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:26.498382    5639 generic.go:356] "Generic (PLEG): container finished" podID="bccf3fd7-7bdd-4e45-9781-6c64d977c616" containerID="1650be9706249db165f5905e674a23e38d4e56658fcdbbb86562b4b7917ee456" exitCode=0
Nov 02 02:11:26.498537 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:26.498476    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee06273b-g8csb" event={"ID":"bccf3fd7-7bdd-4e45-9781-6c64d977c616","Type":"ContainerDied","Data":"1650be9706249db165f5905e674a23e38d4e56658fcdbbb86562b4b7917ee456"}
Nov 02 02:11:26.611959 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:26.611900    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01624761-nqfb5" podStartSLOduration=2.611887099 podStartE2EDuration="2.611887099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:11:26.610171996 +0000 UTC m=+2751.513854109" watchObservedRunningTime="2026-11-02 02:11:26.611887099 +0000 UTC m=+2751.515569213"
Nov 02 02:11:27.218651 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.218614    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b302fb0-288a-4d1d-aece-d87bdf645f92" path="/var/lib/kubelet/pods/5b302fb0-288a-4d1d-aece-d87bdf645f92/volumes"
Nov 02 02:11:27.219035 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.219021    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b145a144-84c0-43c6-b08b-c5c196efddf9" path="/var/lib/kubelet/pods/b145a144-84c0-43c6-b08b-c5c196efddf9/volumes"
Nov 02 02:11:27.421354 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:11:27.421322    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:11:27.507633 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.507542    5639 generic.go:356] "Generic (PLEG): container finished" podID="d97a1388-6ca5-4b25-8a9d-d651faf46527" containerID="a29520a468861a3629622900e57eea8da284fa66ad4cc0409ee70db163086a29" exitCode=0
Nov 02 02:11:27.507633 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.507618    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01624761-nqfb5" event={"ID":"d97a1388-6ca5-4b25-8a9d-d651faf46527","Type":"ContainerDied","Data":"a29520a468861a3629622900e57eea8da284fa66ad4cc0409ee70db163086a29"}
Nov 02 02:11:27.664243 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.664213    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee06273b-g8csb"
Nov 02 02:11:27.701527 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.701499    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38127a13-fkrm6"
Nov 02 02:11:27.746531 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.746485    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" containerID="cri-o://23c946c3e4d5cd35fd8834b077c455df9fde1a6a1ae6d260c7446abe64558f17" gracePeriod=45
Nov 02 02:11:27.757791 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.757725    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main" containerID="cri-o://41abe20bbd71632a17d73c6da7b7538f458b931be2b9bef48b731a29705654e8" gracePeriod=45
Nov 02 02:11:27.791883 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.791841    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/826a3aa6-79b4-4397-9f69-8563b45453e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/826a3aa6-79b4-4397-9f69-8563b45453e2-maas-api-xks-ca\") pod \"826a3aa6-79b4-4397-9f69-8563b45453e2\" (UID: \"826a3aa6-79b4-4397-9f69-8563b45453e2\") "
Nov 02 02:11:27.792054 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.791932    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bccf3fd7-7bdd-4e45-9781-6c64d977c616-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bccf3fd7-7bdd-4e45-9781-6c64d977c616-maas-api-openshift-ca\") pod \"bccf3fd7-7bdd-4e45-9781-6c64d977c616\" (UID: \"bccf3fd7-7bdd-4e45-9781-6c64d977c616\") "
Nov 02 02:11:27.792149 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.792058    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/826a3aa6-79b4-4397-9f69-8563b45453e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/826a3aa6-79b4-4397-9f69-8563b45453e2-maas-api-openshift-ca\") pod \"826a3aa6-79b4-4397-9f69-8563b45453e2\" (UID: \"826a3aa6-79b4-4397-9f69-8563b45453e2\") "
Nov 02 02:11:27.792149 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.792111    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bccf3fd7-7bdd-4e45-9781-6c64d977c616-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bccf3fd7-7bdd-4e45-9781-6c64d977c616-maas-api-xks-ca\") pod \"bccf3fd7-7bdd-4e45-9781-6c64d977c616\" (UID: \"bccf3fd7-7bdd-4e45-9781-6c64d977c616\") "
Nov 02 02:11:27.792262 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.792193    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bccf3fd7-7bdd-4e45-9781-6c64d977c616-kube-api-access-qls27\" (UniqueName: \"kubernetes.io/projected/bccf3fd7-7bdd-4e45-9781-6c64d977c616-kube-api-access-qls27\") pod \"bccf3fd7-7bdd-4e45-9781-6c64d977c616\" (UID: \"bccf3fd7-7bdd-4e45-9781-6c64d977c616\") "
Nov 02 02:11:27.792502 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.792256    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/826a3aa6-79b4-4397-9f69-8563b45453e2-kube-api-access-5t2dz\" (UniqueName: \"kubernetes.io/projected/826a3aa6-79b4-4397-9f69-8563b45453e2-kube-api-access-5t2dz\") pod \"826a3aa6-79b4-4397-9f69-8563b45453e2\" (UID: \"826a3aa6-79b4-4397-9f69-8563b45453e2\") "
Nov 02 02:11:27.794879 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.794834    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/826a3aa6-79b4-4397-9f69-8563b45453e2-maas-api-openshift-ca" pod "826a3aa6-79b4-4397-9f69-8563b45453e2" (UID: "826a3aa6-79b4-4397-9f69-8563b45453e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:11:27.796502 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.796472    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826a3aa6-79b4-4397-9f69-8563b45453e2-kube-api-access-5t2dz" pod "826a3aa6-79b4-4397-9f69-8563b45453e2" (UID: "826a3aa6-79b4-4397-9f69-8563b45453e2"). InnerVolumeSpecName "kube-api-access-5t2dz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:27.796893 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.796865    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bccf3fd7-7bdd-4e45-9781-6c64d977c616-maas-api-openshift-ca" pod "bccf3fd7-7bdd-4e45-9781-6c64d977c616" (UID: "bccf3fd7-7bdd-4e45-9781-6c64d977c616"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:11:27.800435 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.799167    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/826a3aa6-79b4-4397-9f69-8563b45453e2-maas-api-xks-ca" pod "826a3aa6-79b4-4397-9f69-8563b45453e2" (UID: "826a3aa6-79b4-4397-9f69-8563b45453e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:11:27.800435 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.799170    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bccf3fd7-7bdd-4e45-9781-6c64d977c616-kube-api-access-qls27" pod "bccf3fd7-7bdd-4e45-9781-6c64d977c616" (UID: "bccf3fd7-7bdd-4e45-9781-6c64d977c616"). InnerVolumeSpecName "kube-api-access-qls27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:27.800937 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.800909    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bccf3fd7-7bdd-4e45-9781-6c64d977c616-maas-api-xks-ca" pod "bccf3fd7-7bdd-4e45-9781-6c64d977c616" (UID: "bccf3fd7-7bdd-4e45-9781-6c64d977c616"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:11:27.894695 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.894510    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/826a3aa6-79b4-4397-9f69-8563b45453e2-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:27.894695 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.894554    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bccf3fd7-7bdd-4e45-9781-6c64d977c616-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:27.894695 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.894570    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qls27\" (UniqueName: \"kubernetes.io/projected/bccf3fd7-7bdd-4e45-9781-6c64d977c616-kube-api-access-qls27\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:27.894695 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.894585    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t2dz\" (UniqueName: \"kubernetes.io/projected/826a3aa6-79b4-4397-9f69-8563b45453e2-kube-api-access-5t2dz\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:27.894695 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.894600    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/826a3aa6-79b4-4397-9f69-8563b45453e2-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:27.894695 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.894614    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bccf3fd7-7bdd-4e45-9781-6c64d977c616-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:27.908761 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:27.908715    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": dial tcp 10.134.0.59:8000: connect: connection refused"
Nov 02 02:11:28.038900 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.038868    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 02:11:28.041981 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.041957    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 02:11:28.198194 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.198163    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfbf279b-0423-45be-a91c-f35427015145-kube-api-access-gbwnb\" (UniqueName: \"kubernetes.io/projected/dfbf279b-0423-45be-a91c-f35427015145-kube-api-access-gbwnb\") pod \"dfbf279b-0423-45be-a91c-f35427015145\" (UID: \"dfbf279b-0423-45be-a91c-f35427015145\") "
Nov 02 02:11:28.198411 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.198217    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-model-cache\") pod \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\" (UID: \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\") "
Nov 02 02:11:28.198488 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.198469    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b10e1d24-9716-4ff6-89c2-e59452b79a8b-kube-api-access-npm5x\" (UniqueName: \"kubernetes.io/projected/b10e1d24-9716-4ff6-89c2-e59452b79a8b-kube-api-access-npm5x\") pod \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\" (UID: \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\") "
Nov 02 02:11:28.198549 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.198517    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-model-cache\") pod \"dfbf279b-0423-45be-a91c-f35427015145\" (UID: \"dfbf279b-0423-45be-a91c-f35427015145\") "
Nov 02 02:11:28.198611 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.198554    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-home\" (UniqueName: \"kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-home\") pod \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\" (UID: \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\") "
Nov 02 02:11:28.198611 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.198554    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-model-cache" pod "b10e1d24-9716-4ff6-89c2-e59452b79a8b" (UID: "b10e1d24-9716-4ff6-89c2-e59452b79a8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:28.198721 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.198610    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-dshm\") pod \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\" (UID: \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\") "
Nov 02 02:11:28.198721 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.198647    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-tmp-dir\") pod \"dfbf279b-0423-45be-a91c-f35427015145\" (UID: \"dfbf279b-0423-45be-a91c-f35427015145\") "
Nov 02 02:11:28.198721 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.198703    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-dshm\") pod \"dfbf279b-0423-45be-a91c-f35427015145\" (UID: \"dfbf279b-0423-45be-a91c-f35427015145\") "
Nov 02 02:11:28.198881 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.198746    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-tmp-dir\") pod \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\" (UID: \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\") "
Nov 02 02:11:28.198881 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.198789    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfbf279b-0423-45be-a91c-f35427015145-tls-certs\" (UniqueName: \"kubernetes.io/secret/dfbf279b-0423-45be-a91c-f35427015145-tls-certs\") pod \"dfbf279b-0423-45be-a91c-f35427015145\" (UID: \"dfbf279b-0423-45be-a91c-f35427015145\") "
Nov 02 02:11:28.198881 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.198833    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b10e1d24-9716-4ff6-89c2-e59452b79a8b-tls-certs\" (UniqueName: \"kubernetes.io/secret/b10e1d24-9716-4ff6-89c2-e59452b79a8b-tls-certs\") pod \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\" (UID: \"b10e1d24-9716-4ff6-89c2-e59452b79a8b\") "
Nov 02 02:11:28.198881 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.198860    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-home" pod "b10e1d24-9716-4ff6-89c2-e59452b79a8b" (UID: "b10e1d24-9716-4ff6-89c2-e59452b79a8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:28.198881 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.198868    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-model-cache" pod "dfbf279b-0423-45be-a91c-f35427015145" (UID: "dfbf279b-0423-45be-a91c-f35427015145"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:28.199195 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.198879    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-home\" (UniqueName: \"kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-home\") pod \"dfbf279b-0423-45be-a91c-f35427015145\" (UID: \"dfbf279b-0423-45be-a91c-f35427015145\") "
Nov 02 02:11:28.199195 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.199077    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-home" pod "dfbf279b-0423-45be-a91c-f35427015145" (UID: "dfbf279b-0423-45be-a91c-f35427015145"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:28.199195 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.199085    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-tmp-dir" pod "b10e1d24-9716-4ff6-89c2-e59452b79a8b" (UID: "b10e1d24-9716-4ff6-89c2-e59452b79a8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:28.199535 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.199262    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-tmp-dir" pod "dfbf279b-0423-45be-a91c-f35427015145" (UID: "dfbf279b-0423-45be-a91c-f35427015145"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:28.199917 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.199642    5639 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-model-cache\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:28.199917 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.199668    5639 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-model-cache\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:28.199917 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.199683    5639 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-home\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:28.199917 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.199721    5639 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-tmp-dir\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:28.199917 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.199733    5639 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-tmp-dir\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:28.199917 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.199746    5639 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-home\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:28.201353 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.201240    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b10e1d24-9716-4ff6-89c2-e59452b79a8b-kube-api-access-npm5x" pod "b10e1d24-9716-4ff6-89c2-e59452b79a8b" (UID: "b10e1d24-9716-4ff6-89c2-e59452b79a8b"). InnerVolumeSpecName "kube-api-access-npm5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:28.201353 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.201391    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-dshm" pod "b10e1d24-9716-4ff6-89c2-e59452b79a8b" (UID: "b10e1d24-9716-4ff6-89c2-e59452b79a8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:28.201353 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.201410    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfbf279b-0423-45be-a91c-f35427015145-kube-api-access-gbwnb" pod "dfbf279b-0423-45be-a91c-f35427015145" (UID: "dfbf279b-0423-45be-a91c-f35427015145"). InnerVolumeSpecName "kube-api-access-gbwnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:28.201661 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.201642    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfbf279b-0423-45be-a91c-f35427015145-tls-certs" pod "dfbf279b-0423-45be-a91c-f35427015145" (UID: "dfbf279b-0423-45be-a91c-f35427015145"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:11:28.201785 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.201772    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-dshm" pod "dfbf279b-0423-45be-a91c-f35427015145" (UID: "dfbf279b-0423-45be-a91c-f35427015145"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:28.201982 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.201952    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b10e1d24-9716-4ff6-89c2-e59452b79a8b-tls-certs" pod "b10e1d24-9716-4ff6-89c2-e59452b79a8b" (UID: "b10e1d24-9716-4ff6-89c2-e59452b79a8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:11:28.301705 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.301621    5639 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b10e1d24-9716-4ff6-89c2-e59452b79a8b-tls-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:28.301705 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.301661    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbwnb\" (UniqueName: \"kubernetes.io/projected/dfbf279b-0423-45be-a91c-f35427015145-kube-api-access-gbwnb\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:28.301705 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.301677    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npm5x\" (UniqueName: \"kubernetes.io/projected/b10e1d24-9716-4ff6-89c2-e59452b79a8b-kube-api-access-npm5x\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:28.301705 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.301691    5639 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b10e1d24-9716-4ff6-89c2-e59452b79a8b-dshm\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:28.301705 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.301705    5639 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfbf279b-0423-45be-a91c-f35427015145-dshm\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:28.302065 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.301718    5639 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfbf279b-0423-45be-a91c-f35427015145-tls-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:28.515432 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.515397    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38127a13-fkrm6"
Nov 02 02:11:28.515432 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.515406    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38127a13-fkrm6" event={"ID":"826a3aa6-79b4-4397-9f69-8563b45453e2","Type":"ContainerDied","Data":"c4b627a8204fa64a95a65bd37946f5255598748a3d1fe4dfda562a2091df623c"}
Nov 02 02:11:28.515927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.515460    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4b627a8204fa64a95a65bd37946f5255598748a3d1fe4dfda562a2091df623c"
Nov 02 02:11:28.516747 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.516726    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee06273b-g8csb"
Nov 02 02:11:28.516747 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.516733    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee06273b-g8csb" event={"ID":"bccf3fd7-7bdd-4e45-9781-6c64d977c616","Type":"ContainerDied","Data":"1f066848ec8eeb8a68a3dc2731e054f2ca99f7c2edae5b8baca1abe184eb7daa"}
Nov 02 02:11:28.516919 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.516768    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f066848ec8eeb8a68a3dc2731e054f2ca99f7c2edae5b8baca1abe184eb7daa"
Nov 02 02:11:28.518050 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.518026    5639 generic.go:356] "Generic (PLEG): container finished" podID="dfbf279b-0423-45be-a91c-f35427015145" containerID="41abe20bbd71632a17d73c6da7b7538f458b931be2b9bef48b731a29705654e8" exitCode=0
Nov 02 02:11:28.518818 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.518076    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"
Nov 02 02:11:28.518818 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.518114    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" event={"ID":"dfbf279b-0423-45be-a91c-f35427015145","Type":"ContainerDied","Data":"41abe20bbd71632a17d73c6da7b7538f458b931be2b9bef48b731a29705654e8"}
Nov 02 02:11:28.518818 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.518149    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz" event={"ID":"dfbf279b-0423-45be-a91c-f35427015145","Type":"ContainerDied","Data":"9413d2faf8ca66df91275f1bb045f29a2bbbae629f7ecf5675bc965bf8de70f2"}
Nov 02 02:11:28.518818 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.518170    5639 scope.go:122] "RemoveContainer" containerID="41abe20bbd71632a17d73c6da7b7538f458b931be2b9bef48b731a29705654e8"
Nov 02 02:11:28.519902 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.519880    5639 generic.go:356] "Generic (PLEG): container finished" podID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerID="23c946c3e4d5cd35fd8834b077c455df9fde1a6a1ae6d260c7446abe64558f17" exitCode=0
Nov 02 02:11:28.519992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.519918    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"
Nov 02 02:11:28.519992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.519965    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" event={"ID":"b10e1d24-9716-4ff6-89c2-e59452b79a8b","Type":"ContainerDied","Data":"23c946c3e4d5cd35fd8834b077c455df9fde1a6a1ae6d260c7446abe64558f17"}
Nov 02 02:11:28.520107 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.520003    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw" event={"ID":"b10e1d24-9716-4ff6-89c2-e59452b79a8b","Type":"ContainerDied","Data":"0e815115c18a55cf20e90c22522c41e9445ae4168fa229debde6c24f8e3383f9"}
Nov 02 02:11:28.531504 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.531478    5639 scope.go:122] "RemoveContainer" containerID="41abe20bbd71632a17d73c6da7b7538f458b931be2b9bef48b731a29705654e8"
Nov 02 02:11:28.531792 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:11:28.531772    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41abe20bbd71632a17d73c6da7b7538f458b931be2b9bef48b731a29705654e8\": container with ID starting with 41abe20bbd71632a17d73c6da7b7538f458b931be2b9bef48b731a29705654e8 not found: ID does not exist" containerID="41abe20bbd71632a17d73c6da7b7538f458b931be2b9bef48b731a29705654e8"
Nov 02 02:11:28.531871 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.531799    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41abe20bbd71632a17d73c6da7b7538f458b931be2b9bef48b731a29705654e8"} err="failed to get container status \"41abe20bbd71632a17d73c6da7b7538f458b931be2b9bef48b731a29705654e8\": rpc error: code = NotFound desc = could not find container \"41abe20bbd71632a17d73c6da7b7538f458b931be2b9bef48b731a29705654e8\": container with ID starting with 41abe20bbd71632a17d73c6da7b7538f458b931be2b9bef48b731a29705654e8 not found: ID does not exist"
Nov 02 02:11:28.531871 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.531817    5639 scope.go:122] "RemoveContainer" containerID="23c946c3e4d5cd35fd8834b077c455df9fde1a6a1ae6d260c7446abe64558f17"
Nov 02 02:11:28.542486 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.542466    5639 scope.go:122] "RemoveContainer" containerID="23c946c3e4d5cd35fd8834b077c455df9fde1a6a1ae6d260c7446abe64558f17"
Nov 02 02:11:28.542807 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:11:28.542781    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23c946c3e4d5cd35fd8834b077c455df9fde1a6a1ae6d260c7446abe64558f17\": container with ID starting with 23c946c3e4d5cd35fd8834b077c455df9fde1a6a1ae6d260c7446abe64558f17 not found: ID does not exist" containerID="23c946c3e4d5cd35fd8834b077c455df9fde1a6a1ae6d260c7446abe64558f17"
Nov 02 02:11:28.542889 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.542821    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23c946c3e4d5cd35fd8834b077c455df9fde1a6a1ae6d260c7446abe64558f17"} err="failed to get container status \"23c946c3e4d5cd35fd8834b077c455df9fde1a6a1ae6d260c7446abe64558f17\": rpc error: code = NotFound desc = could not find container \"23c946c3e4d5cd35fd8834b077c455df9fde1a6a1ae6d260c7446abe64558f17\": container with ID starting with 23c946c3e4d5cd35fd8834b077c455df9fde1a6a1ae6d260c7446abe64558f17 not found: ID does not exist"
Nov 02 02:11:28.650536 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.650507    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01624761-nqfb5"
Nov 02 02:11:28.733271 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.733236    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"]
Nov 02 02:11:28.737396 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.737367    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-88e76d/simulator-w3-88e76d-kserve-5ccd676f8d-66spz"]
Nov 02 02:11:28.807366 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.807336    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d97a1388-6ca5-4b25-8a9d-d651faf46527-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d97a1388-6ca5-4b25-8a9d-d651faf46527-maas-api-xks-ca\") pod \"d97a1388-6ca5-4b25-8a9d-d651faf46527\" (UID: \"d97a1388-6ca5-4b25-8a9d-d651faf46527\") "
Nov 02 02:11:28.807582 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.807555    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d97a1388-6ca5-4b25-8a9d-d651faf46527-kube-api-access-6lxwd\" (UniqueName: \"kubernetes.io/projected/d97a1388-6ca5-4b25-8a9d-d651faf46527-kube-api-access-6lxwd\") pod \"d97a1388-6ca5-4b25-8a9d-d651faf46527\" (UID: \"d97a1388-6ca5-4b25-8a9d-d651faf46527\") "
Nov 02 02:11:28.807717 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.807642    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d97a1388-6ca5-4b25-8a9d-d651faf46527-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d97a1388-6ca5-4b25-8a9d-d651faf46527-maas-api-openshift-ca\") pod \"d97a1388-6ca5-4b25-8a9d-d651faf46527\" (UID: \"d97a1388-6ca5-4b25-8a9d-d651faf46527\") "
Nov 02 02:11:28.808245 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.808177    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d97a1388-6ca5-4b25-8a9d-d651faf46527-maas-api-openshift-ca" pod "d97a1388-6ca5-4b25-8a9d-d651faf46527" (UID: "d97a1388-6ca5-4b25-8a9d-d651faf46527"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:11:28.809578 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.809549    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d97a1388-6ca5-4b25-8a9d-d651faf46527-maas-api-xks-ca" pod "d97a1388-6ca5-4b25-8a9d-d651faf46527" (UID: "d97a1388-6ca5-4b25-8a9d-d651faf46527"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:11:28.809727 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.809708    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d97a1388-6ca5-4b25-8a9d-d651faf46527-kube-api-access-6lxwd" pod "d97a1388-6ca5-4b25-8a9d-d651faf46527" (UID: "d97a1388-6ca5-4b25-8a9d-d651faf46527"). InnerVolumeSpecName "kube-api-access-6lxwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:28.845521 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.845441    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"]
Nov 02 02:11:28.849265 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.849237    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-88e76d/premium-w3-88e76d-kserve-6b957679cc-gjgjw"]
Nov 02 02:11:28.908938 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.908902    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lxwd\" (UniqueName: \"kubernetes.io/projected/d97a1388-6ca5-4b25-8a9d-d651faf46527-kube-api-access-6lxwd\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:28.908938 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.908931    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d97a1388-6ca5-4b25-8a9d-d651faf46527-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:28.908938 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:28.908942    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d97a1388-6ca5-4b25-8a9d-d651faf46527-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:29.218168 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:29.218082    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" path="/var/lib/kubelet/pods/b10e1d24-9716-4ff6-89c2-e59452b79a8b/volumes"
Nov 02 02:11:29.218779 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:29.218754    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfbf279b-0423-45be-a91c-f35427015145" path="/var/lib/kubelet/pods/dfbf279b-0423-45be-a91c-f35427015145/volumes"
Nov 02 02:11:29.529475 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:29.529381    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01624761-nqfb5" event={"ID":"d97a1388-6ca5-4b25-8a9d-d651faf46527","Type":"ContainerDied","Data":"a6734fbd301afe565e147c27a65cbe2ea13e434963f72749f3ec7580115b47cf"}
Nov 02 02:11:29.529475 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:29.529404    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01624761-nqfb5"
Nov 02 02:11:29.529475 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:29.529443    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6734fbd301afe565e147c27a65cbe2ea13e434963f72749f3ec7580115b47cf"
Nov 02 02:11:38.897649 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:38.897610    5639 scope.go:122] "RemoveContainer" containerID="0fb3aa2d35cd3e6cb4683d05478d8c7dc91306a549116815ba5bc788ff84a05d"
Nov 02 02:11:38.909413 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:38.909385    5639 scope.go:122] "RemoveContainer" containerID="b7d5785f4a52e507254c9584fcc776ed3220b623053aed565caa5a798f481e29"
Nov 02 02:11:40.977064 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:40.977029    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff"]
Nov 02 02:11:41.025826 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:41.025786    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5888dc-7vmff"]
Nov 02 02:11:41.219511 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:41.219474    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="add349ba-4927-476d-ab88-5336942c592f" path="/var/lib/kubelet/pods/add349ba-4927-476d-ab88-5336942c592f/volumes"
Nov 02 02:11:41.455316 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:41.455261    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:11:49.103249 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.103206    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d77bda71-vqg7l"]
Nov 02 02:11:49.103974 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.103951    5639 cpu_manager.go:414] "containerMap: removing container" podUID="bccf3fd7-7bdd-4e45-9781-6c64d977c616" containerName="revoke-keys"
Nov 02 02:11:49.104066 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.103976    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bccf3fd7-7bdd-4e45-9781-6c64d977c616" containerName="revoke-keys"
Nov 02 02:11:49.104066 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.103992    5639 cpu_manager.go:414] "containerMap: removing container" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main"
Nov 02 02:11:49.104066 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.103999    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main"
Nov 02 02:11:49.104066 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.104039    5639 cpu_manager.go:414] "containerMap: removing container" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main"
Nov 02 02:11:49.104066 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.104047    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main"
Nov 02 02:11:49.104066 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.104060    5639 cpu_manager.go:414] "containerMap: removing container" podUID="826a3aa6-79b4-4397-9f69-8563b45453e2" containerName="revoke-keys"
Nov 02 02:11:49.104066 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.104067    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="826a3aa6-79b4-4397-9f69-8563b45453e2" containerName="revoke-keys"
Nov 02 02:11:49.104478 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.104089    5639 cpu_manager.go:414] "containerMap: removing container" podUID="b145a144-84c0-43c6-b08b-c5c196efddf9" containerName="payload-pre-processing"
Nov 02 02:11:49.104478 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.104097    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b145a144-84c0-43c6-b08b-c5c196efddf9" containerName="payload-pre-processing"
Nov 02 02:11:49.104478 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.104116    5639 cpu_manager.go:414] "containerMap: removing container" podUID="d97a1388-6ca5-4b25-8a9d-d651faf46527" containerName="revoke-keys"
Nov 02 02:11:49.104478 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.104123    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d97a1388-6ca5-4b25-8a9d-d651faf46527" containerName="revoke-keys"
Nov 02 02:11:49.104478 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.104258    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="b10e1d24-9716-4ff6-89c2-e59452b79a8b" containerName="main"
Nov 02 02:11:49.104478 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.104273    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="826a3aa6-79b4-4397-9f69-8563b45453e2" containerName="revoke-keys"
Nov 02 02:11:49.104478 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.104324    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="b145a144-84c0-43c6-b08b-c5c196efddf9" containerName="payload-pre-processing"
Nov 02 02:11:49.104478 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.104337    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="d97a1388-6ca5-4b25-8a9d-d651faf46527" containerName="revoke-keys"
Nov 02 02:11:49.104478 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.104351    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="bccf3fd7-7bdd-4e45-9781-6c64d977c616" containerName="revoke-keys"
Nov 02 02:11:49.104478 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.104360    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfbf279b-0423-45be-a91c-f35427015145" containerName="main"
Nov 02 02:11:49.109652 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.109625    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d77bda71-vqg7l"
Nov 02 02:11:49.161976 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.161800    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:11:49.211341 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.211303    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:11:49.211641 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.211617    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:11:49.211771 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.211645    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:11:49.218131 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.218102    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d77bda71-vqg7l"]
Nov 02 02:11:49.278795 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.278751    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6gkp\" (UniqueName: \"kubernetes.io/projected/fdb9bf00-7cc7-4476-a8f5-53aab6d658e0-kube-api-access-t6gkp\") pod \"maas-api-revoke-sub-d77bda71-vqg7l\" (UID: \"fdb9bf00-7cc7-4476-a8f5-53aab6d658e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d77bda71-vqg7l"
Nov 02 02:11:49.279017 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.278985    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdb9bf00-7cc7-4476-a8f5-53aab6d658e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d77bda71-vqg7l\" (UID: \"fdb9bf00-7cc7-4476-a8f5-53aab6d658e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d77bda71-vqg7l"
Nov 02 02:11:49.279099 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.279038    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdb9bf00-7cc7-4476-a8f5-53aab6d658e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d77bda71-vqg7l\" (UID: \"fdb9bf00-7cc7-4476-a8f5-53aab6d658e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d77bda71-vqg7l"
Nov 02 02:11:49.380820 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.380712    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdb9bf00-7cc7-4476-a8f5-53aab6d658e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d77bda71-vqg7l\" (UID: \"fdb9bf00-7cc7-4476-a8f5-53aab6d658e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d77bda71-vqg7l"
Nov 02 02:11:49.380820 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.380768    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdb9bf00-7cc7-4476-a8f5-53aab6d658e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d77bda71-vqg7l\" (UID: \"fdb9bf00-7cc7-4476-a8f5-53aab6d658e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d77bda71-vqg7l"
Nov 02 02:11:49.381071 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.380847    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6gkp\" (UniqueName: \"kubernetes.io/projected/fdb9bf00-7cc7-4476-a8f5-53aab6d658e0-kube-api-access-t6gkp\") pod \"maas-api-revoke-sub-d77bda71-vqg7l\" (UID: \"fdb9bf00-7cc7-4476-a8f5-53aab6d658e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d77bda71-vqg7l"
Nov 02 02:11:49.381538 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.381516    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdb9bf00-7cc7-4476-a8f5-53aab6d658e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d77bda71-vqg7l\" (UID: \"fdb9bf00-7cc7-4476-a8f5-53aab6d658e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d77bda71-vqg7l"
Nov 02 02:11:49.383396 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.383378    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdb9bf00-7cc7-4476-a8f5-53aab6d658e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d77bda71-vqg7l\" (UID: \"fdb9bf00-7cc7-4476-a8f5-53aab6d658e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d77bda71-vqg7l"
Nov 02 02:11:49.442169 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.442131    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6gkp\" (UniqueName: \"kubernetes.io/projected/fdb9bf00-7cc7-4476-a8f5-53aab6d658e0-kube-api-access-t6gkp\") pod \"maas-api-revoke-sub-d77bda71-vqg7l\" (UID: \"fdb9bf00-7cc7-4476-a8f5-53aab6d658e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d77bda71-vqg7l"
Nov 02 02:11:49.723869 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:49.723765    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d77bda71-vqg7l"
Nov 02 02:11:50.101540 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:50.101511    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d77bda71-vqg7l"]
Nov 02 02:11:50.103209 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:11:50.103181    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb9bf00_7cc7_4476_a8f5_53aab6d658e0.slice/crio-2f05ebd2b64a4bdb48a2010da3738d04af6f89387bafb012c0b57c69acbff1bd WatchSource:0}: Error finding container 2f05ebd2b64a4bdb48a2010da3738d04af6f89387bafb012c0b57c69acbff1bd: Status 404 returned error can't find the container with id 2f05ebd2b64a4bdb48a2010da3738d04af6f89387bafb012c0b57c69acbff1bd
Nov 02 02:11:50.688363 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:50.688233    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d77bda71-vqg7l" event={"ID":"fdb9bf00-7cc7-4476-a8f5-53aab6d658e0","Type":"ContainerStarted","Data":"a8a04ef230adbedb2f4cbec71b54181a1bc30e4feb3566fc43dfc1f206cfbc93"}
Nov 02 02:11:50.688363 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:50.688329    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d77bda71-vqg7l" event={"ID":"fdb9bf00-7cc7-4476-a8f5-53aab6d658e0","Type":"ContainerStarted","Data":"2f05ebd2b64a4bdb48a2010da3738d04af6f89387bafb012c0b57c69acbff1bd"}
Nov 02 02:11:50.833741 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:50.833690    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d77bda71-vqg7l" podStartSLOduration=1.833674185 podStartE2EDuration="1.833674185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:11:50.831464532 +0000 UTC m=+2775.735146645" watchObservedRunningTime="2026-11-02 02:11:50.833674185 +0000 UTC m=+2775.737356302"
Nov 02 02:11:51.695732 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:51.695700    5639 generic.go:356] "Generic (PLEG): container finished" podID="fdb9bf00-7cc7-4476-a8f5-53aab6d658e0" containerID="a8a04ef230adbedb2f4cbec71b54181a1bc30e4feb3566fc43dfc1f206cfbc93" exitCode=0
Nov 02 02:11:51.696131 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:51.695744    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d77bda71-vqg7l" event={"ID":"fdb9bf00-7cc7-4476-a8f5-53aab6d658e0","Type":"ContainerDied","Data":"a8a04ef230adbedb2f4cbec71b54181a1bc30e4feb3566fc43dfc1f206cfbc93"}
Nov 02 02:11:52.853732 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:52.853704    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d77bda71-vqg7l"
Nov 02 02:11:53.023555 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:53.023470    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdb9bf00-7cc7-4476-a8f5-53aab6d658e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdb9bf00-7cc7-4476-a8f5-53aab6d658e0-maas-api-openshift-ca\") pod \"fdb9bf00-7cc7-4476-a8f5-53aab6d658e0\" (UID: \"fdb9bf00-7cc7-4476-a8f5-53aab6d658e0\") "
Nov 02 02:11:53.023555 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:53.023515    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdb9bf00-7cc7-4476-a8f5-53aab6d658e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdb9bf00-7cc7-4476-a8f5-53aab6d658e0-maas-api-xks-ca\") pod \"fdb9bf00-7cc7-4476-a8f5-53aab6d658e0\" (UID: \"fdb9bf00-7cc7-4476-a8f5-53aab6d658e0\") "
Nov 02 02:11:53.023818 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:53.023602    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb9bf00-7cc7-4476-a8f5-53aab6d658e0-kube-api-access-t6gkp\" (UniqueName: \"kubernetes.io/projected/fdb9bf00-7cc7-4476-a8f5-53aab6d658e0-kube-api-access-t6gkp\") pod \"fdb9bf00-7cc7-4476-a8f5-53aab6d658e0\" (UID: \"fdb9bf00-7cc7-4476-a8f5-53aab6d658e0\") "
Nov 02 02:11:53.023879 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:53.023825    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb9bf00-7cc7-4476-a8f5-53aab6d658e0-maas-api-openshift-ca" pod "fdb9bf00-7cc7-4476-a8f5-53aab6d658e0" (UID: "fdb9bf00-7cc7-4476-a8f5-53aab6d658e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:11:53.024124 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:53.024100    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdb9bf00-7cc7-4476-a8f5-53aab6d658e0-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:53.025938 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:53.025912    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdb9bf00-7cc7-4476-a8f5-53aab6d658e0-maas-api-xks-ca" pod "fdb9bf00-7cc7-4476-a8f5-53aab6d658e0" (UID: "fdb9bf00-7cc7-4476-a8f5-53aab6d658e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:11:53.025938 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:53.025931    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb9bf00-7cc7-4476-a8f5-53aab6d658e0-kube-api-access-t6gkp" pod "fdb9bf00-7cc7-4476-a8f5-53aab6d658e0" (UID: "fdb9bf00-7cc7-4476-a8f5-53aab6d658e0"). InnerVolumeSpecName "kube-api-access-t6gkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:53.125665 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:53.125621    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdb9bf00-7cc7-4476-a8f5-53aab6d658e0-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:53.125665 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:53.125656    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6gkp\" (UniqueName: \"kubernetes.io/projected/fdb9bf00-7cc7-4476-a8f5-53aab6d658e0-kube-api-access-t6gkp\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:53.709468 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:53.709436    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d77bda71-vqg7l"
Nov 02 02:11:53.709629 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:53.709466    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d77bda71-vqg7l" event={"ID":"fdb9bf00-7cc7-4476-a8f5-53aab6d658e0","Type":"ContainerDied","Data":"2f05ebd2b64a4bdb48a2010da3738d04af6f89387bafb012c0b57c69acbff1bd"}
Nov 02 02:11:53.709629 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:53.709531    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f05ebd2b64a4bdb48a2010da3738d04af6f89387bafb012c0b57c69acbff1bd"
Nov 02 02:11:57.421608 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:11:57.421572    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:11:59.048827 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:59.048788    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4732f101-w6z8r"]
Nov 02 02:11:59.098411 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:59.098375    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4732f101-w6z8r"]
Nov 02 02:11:59.217696 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:11:59.217660    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965" path="/var/lib/kubelet/pods/f392f4c7-5ef3-4f2f-a7f8-d0ad1d90d965/volumes"
Nov 02 02:12:01.631575 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:01.631535    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd"]
Nov 02 02:12:01.681122 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:01.681088    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88e76d-vcttd"]
Nov 02 02:12:03.219658 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:03.219617    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8f6bc15-77ce-4b46-9a01-106f2518402a" path="/var/lib/kubelet/pods/d8f6bc15-77ce-4b46-9a01-106f2518402a/volumes"
Nov 02 02:12:10.338157 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:10.338111    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t"]
Nov 02 02:12:10.340421 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:10.338735    5639 cpu_manager.go:414] "containerMap: removing container" podUID="fdb9bf00-7cc7-4476-a8f5-53aab6d658e0" containerName="revoke-keys"
Nov 02 02:12:10.340421 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:10.338752    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb9bf00-7cc7-4476-a8f5-53aab6d658e0" containerName="revoke-keys"
Nov 02 02:12:10.340421 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:10.338843    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb9bf00-7cc7-4476-a8f5-53aab6d658e0" containerName="revoke-keys"
Nov 02 02:12:10.342002 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:10.341985    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t"
Nov 02 02:12:10.397419 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:10.397383    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:12:10.425588 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:10.425553    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t\" (UID: \"3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t"
Nov 02 02:12:10.425794 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:10.425686    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t\" (UID: \"3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t"
Nov 02 02:12:10.425794 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:10.425772    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l5dk\" (UniqueName: \"kubernetes.io/projected/3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71-kube-api-access-2l5dk\") pod \"maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t\" (UID: \"3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t"
Nov 02 02:12:10.446489 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:10.446450    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:12:10.446489 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:10.446461    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:12:10.446702 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:10.446507    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:12:10.450697 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:10.450655    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t"]
Nov 02 02:12:10.526921 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:10.526884    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t\" (UID: \"3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t"
Nov 02 02:12:10.527117 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:10.526950    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t\" (UID: \"3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t"
Nov 02 02:12:10.527117 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:10.527009    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l5dk\" (UniqueName: \"kubernetes.io/projected/3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71-kube-api-access-2l5dk\") pod \"maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t\" (UID: \"3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t"
Nov 02 02:12:10.527732 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:10.527705    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t\" (UID: \"3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t"
Nov 02 02:12:10.529408 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:10.529387    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t\" (UID: \"3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t"
Nov 02 02:12:10.588743 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:10.588665    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l5dk\" (UniqueName: \"kubernetes.io/projected/3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71-kube-api-access-2l5dk\") pod \"maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t\" (UID: \"3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t"
Nov 02 02:12:10.656018 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:10.655984    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t"
Nov 02 02:12:11.032558 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:11.032525    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t"]
Nov 02 02:12:11.034918 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:12:11.034889    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c7fb5f1_8ba2_4ece_acb5_fbc48f375a71.slice/crio-9e2143744ee7aff7adb715587ee6a7bea6bd70c94a0d88c44bcc7fca3530ed98 WatchSource:0}: Error finding container 9e2143744ee7aff7adb715587ee6a7bea6bd70c94a0d88c44bcc7fca3530ed98: Status 404 returned error can't find the container with id 9e2143744ee7aff7adb715587ee6a7bea6bd70c94a0d88c44bcc7fca3530ed98
Nov 02 02:12:11.838086 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:11.838049    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t" event={"ID":"3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71","Type":"ContainerStarted","Data":"da713d436c5490b5e1990e1f40edcdc40d4113a2775cd2ed615dd6ed97bf2d2d"}
Nov 02 02:12:11.838548 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:11.838111    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t" event={"ID":"3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71","Type":"ContainerStarted","Data":"9e2143744ee7aff7adb715587ee6a7bea6bd70c94a0d88c44bcc7fca3530ed98"}
Nov 02 02:12:11.955406 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:11.955349    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t" podStartSLOduration=1.9553343060000001 podStartE2EDuration="1.955334306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:12:11.953374868 +0000 UTC m=+2796.857056981" watchObservedRunningTime="2026-11-02 02:12:11.955334306 +0000 UTC m=+2796.859016440"
Nov 02 02:12:12.846163 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:12.846131    5639 generic.go:356] "Generic (PLEG): container finished" podID="3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71" containerID="da713d436c5490b5e1990e1f40edcdc40d4113a2775cd2ed615dd6ed97bf2d2d" exitCode=0
Nov 02 02:12:12.846656 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:12.846229    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t" event={"ID":"3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71","Type":"ContainerDied","Data":"da713d436c5490b5e1990e1f40edcdc40d4113a2775cd2ed615dd6ed97bf2d2d"}
Nov 02 02:12:13.993813 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:13.993781    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t"
Nov 02 02:12:14.070709 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:14.070654    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71-maas-api-openshift-ca\") pod \"3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71\" (UID: \"3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71\") "
Nov 02 02:12:14.070925 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:14.070758    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71-maas-api-xks-ca\") pod \"3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71\" (UID: \"3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71\") "
Nov 02 02:12:14.070925 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:14.070827    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71-kube-api-access-2l5dk\" (UniqueName: \"kubernetes.io/projected/3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71-kube-api-access-2l5dk\") pod \"3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71\" (UID: \"3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71\") "
Nov 02 02:12:14.071145 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:14.071114    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71-maas-api-openshift-ca" pod "3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71" (UID: "3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:12:14.071272 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:14.071249    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:14.073413 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:14.073382    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71-maas-api-xks-ca" pod "3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71" (UID: "3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:12:14.073413 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:14.073405    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71-kube-api-access-2l5dk" pod "3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71" (UID: "3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71"). InnerVolumeSpecName "kube-api-access-2l5dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:12:14.172142 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:14.172058    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:14.172142 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:14.172091    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l5dk\" (UniqueName: \"kubernetes.io/projected/3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71-kube-api-access-2l5dk\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:14.415209 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:14.415170    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:12:14.862003 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:14.861975    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t"
Nov 02 02:12:14.862202 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:14.861973    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t" event={"ID":"3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71","Type":"ContainerDied","Data":"9e2143744ee7aff7adb715587ee6a7bea6bd70c94a0d88c44bcc7fca3530ed98"}
Nov 02 02:12:14.862202 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:14.862101    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e2143744ee7aff7adb715587ee6a7bea6bd70c94a0d88c44bcc7fca3530ed98"
Nov 02 02:12:21.818903 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:21.818814    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj"]
Nov 02 02:12:26.819383 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:26.819339    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj" podUID="585317a1-6f66-4c71-88a5-bb3089999ff7" containerName="payload-processing" containerID="cri-o://69db31dd89634c73a589993b8fbe43ad52a8f92da2054780a4dc2bb228a32b8e" gracePeriod=25
Nov 02 02:12:26.940689 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:26.940661    5639 generic.go:356] "Generic (PLEG): container finished" podID="585317a1-6f66-4c71-88a5-bb3089999ff7" containerID="69db31dd89634c73a589993b8fbe43ad52a8f92da2054780a4dc2bb228a32b8e" exitCode=0
Nov 02 02:12:26.940808 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:26.940735    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj" event={"ID":"585317a1-6f66-4c71-88a5-bb3089999ff7","Type":"ContainerDied","Data":"69db31dd89634c73a589993b8fbe43ad52a8f92da2054780a4dc2bb228a32b8e"}
Nov 02 02:12:27.076237 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:27.076179    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj"
Nov 02 02:12:27.108456 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:27.108428    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/585317a1-6f66-4c71-88a5-bb3089999ff7-kube-api-access-znndn\" (UniqueName: \"kubernetes.io/projected/585317a1-6f66-4c71-88a5-bb3089999ff7-kube-api-access-znndn\") pod \"585317a1-6f66-4c71-88a5-bb3089999ff7\" (UID: \"585317a1-6f66-4c71-88a5-bb3089999ff7\") "
Nov 02 02:12:27.108623 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:27.108607    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/585317a1-6f66-4c71-88a5-bb3089999ff7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/585317a1-6f66-4c71-88a5-bb3089999ff7-plugins-config-volume\") pod \"585317a1-6f66-4c71-88a5-bb3089999ff7\" (UID: \"585317a1-6f66-4c71-88a5-bb3089999ff7\") "
Nov 02 02:12:27.108960 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:27.108934    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/585317a1-6f66-4c71-88a5-bb3089999ff7-plugins-config-volume" pod "585317a1-6f66-4c71-88a5-bb3089999ff7" (UID: "585317a1-6f66-4c71-88a5-bb3089999ff7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:12:27.110684 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:27.110664    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/585317a1-6f66-4c71-88a5-bb3089999ff7-kube-api-access-znndn" pod "585317a1-6f66-4c71-88a5-bb3089999ff7" (UID: "585317a1-6f66-4c71-88a5-bb3089999ff7"). InnerVolumeSpecName "kube-api-access-znndn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:12:27.209841 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:27.209812    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znndn\" (UniqueName: \"kubernetes.io/projected/585317a1-6f66-4c71-88a5-bb3089999ff7-kube-api-access-znndn\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:27.209841 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:27.209839    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/585317a1-6f66-4c71-88a5-bb3089999ff7-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:27.420803 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:12:27.420756    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:12:27.948228 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:27.948195    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj" event={"ID":"585317a1-6f66-4c71-88a5-bb3089999ff7","Type":"ContainerDied","Data":"0368a3ad9fd6298dda14082cf8b1e566c4e3133fa522402878146046cd0c6a63"}
Nov 02 02:12:27.948749 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:27.948248    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj"
Nov 02 02:12:27.948749 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:27.948297    5639 scope.go:122] "RemoveContainer" containerID="69db31dd89634c73a589993b8fbe43ad52a8f92da2054780a4dc2bb228a32b8e"
Nov 02 02:12:28.115002 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:28.114968    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj"]
Nov 02 02:12:28.164543 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:28.164509    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d0fdb9-f8f7df887-6b9zj"]
Nov 02 02:12:29.217799 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:29.217766    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="585317a1-6f66-4c71-88a5-bb3089999ff7" path="/var/lib/kubelet/pods/585317a1-6f66-4c71-88a5-bb3089999ff7/volumes"
Nov 02 02:12:35.048224 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:35.048183    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63995e36-trjk5"]
Nov 02 02:12:35.098175 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:35.098116    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6bebaca-f2l77"]
Nov 02 02:12:35.100292 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:35.100250    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63995e36-trjk5"]
Nov 02 02:12:35.102094 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:35.102068    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6bebaca-f2l77"]
Nov 02 02:12:35.218017 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:35.217985    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="321ead4c-a53e-4ff3-9443-2c076c91b7fc" path="/var/lib/kubelet/pods/321ead4c-a53e-4ff3-9443-2c076c91b7fc/volumes"
Nov 02 02:12:35.218434 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:35.218419    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f606ebbc-224b-457e-a324-6bfcf27c6a1b" path="/var/lib/kubelet/pods/f606ebbc-224b-457e-a324-6bfcf27c6a1b/volumes"
Nov 02 02:12:46.854920 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:46.854883    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t"]
Nov 02 02:12:46.904202 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:46.904166    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0fdb9-kzf7t"]
Nov 02 02:12:47.217658 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:47.217582    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71" path="/var/lib/kubelet/pods/3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71/volumes"
Nov 02 02:12:50.732465 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:50.732429    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:12:53.013691 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:53.013658    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"]
Nov 02 02:12:53.014147 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:53.013868    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh" podUID="389a930e-b478-448a-a3e6-a3466a362c22" containerName="istio-proxy" containerID="cri-o://b0ed4f58003e70fb22a6d62908ee68a7cc3dac080f1655cda0823518dc4e5521" gracePeriod=30
Nov 02 02:12:53.062467 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:53.062432    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"]
Nov 02 02:12:53.062654 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:53.062632    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn" podUID="6dbfefd1-0270-4d23-be4b-b4f3561e877e" containerName="istio-proxy" containerID="cri-o://12a4cadebcebad0565852db61fa3df34f6944343b6b7cdaf45355402f12da42e" gracePeriod=30
Nov 02 02:12:54.110094 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:54.110049    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"]
Nov 02 02:12:54.110698 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:54.110298    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5" podUID="0c89cdf8-4ee0-444d-a8a3-8371f375c432" containerName="istio-proxy" containerID="cri-o://d7b9e5030228b7b7a01446d7629759ee09f38d1df8df035055f4ee3603f2101d" gracePeriod=30
Nov 02 02:12:55.008552 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:55.008520    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn"]
Nov 02 02:12:55.009040 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:55.009028    5639 cpu_manager.go:414] "containerMap: removing container" podUID="3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71" containerName="revoke-keys"
Nov 02 02:12:55.009092 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:55.009042    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71" containerName="revoke-keys"
Nov 02 02:12:55.009092 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:55.009064    5639 cpu_manager.go:414] "containerMap: removing container" podUID="585317a1-6f66-4c71-88a5-bb3089999ff7" containerName="payload-processing"
Nov 02 02:12:55.009092 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:55.009070    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="585317a1-6f66-4c71-88a5-bb3089999ff7" containerName="payload-processing"
Nov 02 02:12:55.009194 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:55.009147    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="585317a1-6f66-4c71-88a5-bb3089999ff7" containerName="payload-processing"
Nov 02 02:12:55.009194 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:55.009162    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c7fb5f1-8ba2-4ece-acb5-fbc48f375a71" containerName="revoke-keys"
Nov 02 02:12:55.013919 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:55.013897    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn"
Nov 02 02:12:55.039573 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:55.039547    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a24e870c-527e-43a4-9eb7-337b49468d84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn\" (UID: \"a24e870c-527e-43a4-9eb7-337b49468d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn"
Nov 02 02:12:55.039716 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:55.039606    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k8sr\" (UniqueName: \"kubernetes.io/projected/a24e870c-527e-43a4-9eb7-337b49468d84-kube-api-access-9k8sr\") pod \"maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn\" (UID: \"a24e870c-527e-43a4-9eb7-337b49468d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn"
Nov 02 02:12:55.039768 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:55.039747    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a24e870c-527e-43a4-9eb7-337b49468d84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn\" (UID: \"a24e870c-527e-43a4-9eb7-337b49468d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn"
Nov 02 02:12:55.116541 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:55.116510    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:12:55.116541 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:55.116526    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:12:55.116956 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:55.116603    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:12:55.116956 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:55.116639    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:12:55.118940 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:55.118917    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn"]
Nov 02 02:12:55.140551 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:55.140527    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a24e870c-527e-43a4-9eb7-337b49468d84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn\" (UID: \"a24e870c-527e-43a4-9eb7-337b49468d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn"
Nov 02 02:12:55.140719 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:55.140603    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a24e870c-527e-43a4-9eb7-337b49468d84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn\" (UID: \"a24e870c-527e-43a4-9eb7-337b49468d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn"
Nov 02 02:12:55.140719 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:55.140651    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k8sr\" (UniqueName: \"kubernetes.io/projected/a24e870c-527e-43a4-9eb7-337b49468d84-kube-api-access-9k8sr\") pod \"maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn\" (UID: \"a24e870c-527e-43a4-9eb7-337b49468d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn"
Nov 02 02:12:55.141333 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:55.141254    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a24e870c-527e-43a4-9eb7-337b49468d84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn\" (UID: \"a24e870c-527e-43a4-9eb7-337b49468d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn"
Nov 02 02:12:55.143025 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:55.143004    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a24e870c-527e-43a4-9eb7-337b49468d84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn\" (UID: \"a24e870c-527e-43a4-9eb7-337b49468d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn"
Nov 02 02:12:55.201687 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:55.201659    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k8sr\" (UniqueName: \"kubernetes.io/projected/a24e870c-527e-43a4-9eb7-337b49468d84-kube-api-access-9k8sr\") pod \"maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn\" (UID: \"a24e870c-527e-43a4-9eb7-337b49468d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn"
Nov 02 02:12:55.327019 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:55.326936    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn"
Nov 02 02:12:55.702292 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:55.702243    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn"]
Nov 02 02:12:55.703712 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:12:55.703682    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda24e870c_527e_43a4_9eb7_337b49468d84.slice/crio-62287779bf4daab74140c2eb93d2363d134dcf3d01f96a89b382b6036b50fff1 WatchSource:0}: Error finding container 62287779bf4daab74140c2eb93d2363d134dcf3d01f96a89b382b6036b50fff1: Status 404 returned error can't find the container with id 62287779bf4daab74140c2eb93d2363d134dcf3d01f96a89b382b6036b50fff1
Nov 02 02:12:56.125122 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:56.125088    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn" event={"ID":"a24e870c-527e-43a4-9eb7-337b49468d84","Type":"ContainerStarted","Data":"46b85f8b26bb4eb939a030fc1bf3048c47bf0a15b86d374d7721df0ebe14e144"}
Nov 02 02:12:56.125594 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:56.125149    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn" event={"ID":"a24e870c-527e-43a4-9eb7-337b49468d84","Type":"ContainerStarted","Data":"62287779bf4daab74140c2eb93d2363d134dcf3d01f96a89b382b6036b50fff1"}
Nov 02 02:12:56.243367 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:56.243233    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn" podStartSLOduration=2.243214297 podStartE2EDuration="2.243214297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:12:56.240823741 +0000 UTC m=+2841.144505854" watchObservedRunningTime="2026-11-02 02:12:56.243214297 +0000 UTC m=+2841.146896412"
Nov 02 02:12:57.132620 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:57.132580    5639 generic.go:356] "Generic (PLEG): container finished" podID="a24e870c-527e-43a4-9eb7-337b49468d84" containerID="46b85f8b26bb4eb939a030fc1bf3048c47bf0a15b86d374d7721df0ebe14e144" exitCode=0
Nov 02 02:12:57.133027 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:57.132640    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn" event={"ID":"a24e870c-527e-43a4-9eb7-337b49468d84","Type":"ContainerDied","Data":"46b85f8b26bb4eb939a030fc1bf3048c47bf0a15b86d374d7721df0ebe14e144"}
Nov 02 02:12:57.420244 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:12:57.420164    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:12:58.140068 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.140037    5639 generic.go:356] "Generic (PLEG): container finished" podID="389a930e-b478-448a-a3e6-a3466a362c22" containerID="b0ed4f58003e70fb22a6d62908ee68a7cc3dac080f1655cda0823518dc4e5521" exitCode=0
Nov 02 02:12:58.140492 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.140113    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh" event={"ID":"389a930e-b478-448a-a3e6-a3466a362c22","Type":"ContainerDied","Data":"b0ed4f58003e70fb22a6d62908ee68a7cc3dac080f1655cda0823518dc4e5521"}
Nov 02 02:12:58.289425 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.289218    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn"
Nov 02 02:12:58.339815 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.339786    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:12:58.358709 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.358689    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:12:58.377813 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.377789    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a24e870c-527e-43a4-9eb7-337b49468d84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a24e870c-527e-43a4-9eb7-337b49468d84-maas-api-xks-ca\") pod \"a24e870c-527e-43a4-9eb7-337b49468d84\" (UID: \"a24e870c-527e-43a4-9eb7-337b49468d84\") "
Nov 02 02:12:58.377959 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.377823    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-istio-data\") pod \"389a930e-b478-448a-a3e6-a3466a362c22\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") "
Nov 02 02:12:58.377959 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.377842    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-workload-certs\") pod \"389a930e-b478-448a-a3e6-a3466a362c22\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") "
Nov 02 02:12:58.377959 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.377880    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-workload-socket\") pod \"389a930e-b478-448a-a3e6-a3466a362c22\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") "
Nov 02 02:12:58.377959 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.377914    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-credential-socket\") pod \"389a930e-b478-448a-a3e6-a3466a362c22\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") "
Nov 02 02:12:58.377959 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.377932    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/389a930e-b478-448a-a3e6-a3466a362c22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389a930e-b478-448a-a3e6-a3466a362c22-istiod-ca-cert\") pod \"389a930e-b478-448a-a3e6-a3466a362c22\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") "
Nov 02 02:12:58.378251 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.377963    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a24e870c-527e-43a4-9eb7-337b49468d84-kube-api-access-9k8sr\" (UniqueName: \"kubernetes.io/projected/a24e870c-527e-43a4-9eb7-337b49468d84-kube-api-access-9k8sr\") pod \"a24e870c-527e-43a4-9eb7-337b49468d84\" (UID: \"a24e870c-527e-43a4-9eb7-337b49468d84\") "
Nov 02 02:12:58.378251 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.377998    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-istio-envoy\") pod \"389a930e-b478-448a-a3e6-a3466a362c22\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") "
Nov 02 02:12:58.378251 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.378034    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/389a930e-b478-448a-a3e6-a3466a362c22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389a930e-b478-448a-a3e6-a3466a362c22-istio-podinfo\") pod \"389a930e-b478-448a-a3e6-a3466a362c22\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") "
Nov 02 02:12:58.378251 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.378089    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/389a930e-b478-448a-a3e6-a3466a362c22-istio-token\" (UniqueName: \"kubernetes.io/projected/389a930e-b478-448a-a3e6-a3466a362c22-istio-token\") pod \"389a930e-b478-448a-a3e6-a3466a362c22\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") "
Nov 02 02:12:58.378251 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.378131    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/389a930e-b478-448a-a3e6-a3466a362c22-kube-api-access-jzc4d\" (UniqueName: \"kubernetes.io/projected/389a930e-b478-448a-a3e6-a3466a362c22-kube-api-access-jzc4d\") pod \"389a930e-b478-448a-a3e6-a3466a362c22\" (UID: \"389a930e-b478-448a-a3e6-a3466a362c22\") "
Nov 02 02:12:58.378251 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.378121    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-workload-socket" pod "389a930e-b478-448a-a3e6-a3466a362c22" (UID: "389a930e-b478-448a-a3e6-a3466a362c22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:58.378251 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.378164    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-istio-data" pod "389a930e-b478-448a-a3e6-a3466a362c22" (UID: "389a930e-b478-448a-a3e6-a3466a362c22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:58.378251 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.378180    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a24e870c-527e-43a4-9eb7-337b49468d84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a24e870c-527e-43a4-9eb7-337b49468d84-maas-api-openshift-ca\") pod \"a24e870c-527e-43a4-9eb7-337b49468d84\" (UID: \"a24e870c-527e-43a4-9eb7-337b49468d84\") "
Nov 02 02:12:58.378251 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.378189    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-workload-certs" pod "389a930e-b478-448a-a3e6-a3466a362c22" (UID: "389a930e-b478-448a-a3e6-a3466a362c22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:58.378948 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.378323    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-credential-socket" pod "389a930e-b478-448a-a3e6-a3466a362c22" (UID: "389a930e-b478-448a-a3e6-a3466a362c22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:58.378948 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.378682    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:58.378948 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.378703    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:58.378948 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.378717    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:58.378948 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.378732    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:58.378948 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.378765    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/389a930e-b478-448a-a3e6-a3466a362c22-istiod-ca-cert" pod "389a930e-b478-448a-a3e6-a3466a362c22" (UID: "389a930e-b478-448a-a3e6-a3466a362c22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:12:58.379317 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.378998    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a24e870c-527e-43a4-9eb7-337b49468d84-maas-api-openshift-ca" pod "a24e870c-527e-43a4-9eb7-337b49468d84" (UID: "a24e870c-527e-43a4-9eb7-337b49468d84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:12:58.380631 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.380598    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389a930e-b478-448a-a3e6-a3466a362c22-istio-token" pod "389a930e-b478-448a-a3e6-a3466a362c22" (UID: "389a930e-b478-448a-a3e6-a3466a362c22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:12:58.380732 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.380632    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/389a930e-b478-448a-a3e6-a3466a362c22-istio-podinfo" pod "389a930e-b478-448a-a3e6-a3466a362c22" (UID: "389a930e-b478-448a-a3e6-a3466a362c22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:12:58.380732 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.380721    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a24e870c-527e-43a4-9eb7-337b49468d84-maas-api-xks-ca" pod "a24e870c-527e-43a4-9eb7-337b49468d84" (UID: "a24e870c-527e-43a4-9eb7-337b49468d84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:12:58.380835 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.380820    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389a930e-b478-448a-a3e6-a3466a362c22-kube-api-access-jzc4d" pod "389a930e-b478-448a-a3e6-a3466a362c22" (UID: "389a930e-b478-448a-a3e6-a3466a362c22"). InnerVolumeSpecName "kube-api-access-jzc4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:12:58.381068 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.381049    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-istio-envoy" pod "389a930e-b478-448a-a3e6-a3466a362c22" (UID: "389a930e-b478-448a-a3e6-a3466a362c22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:58.381183 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.381167    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a24e870c-527e-43a4-9eb7-337b49468d84-kube-api-access-9k8sr" pod "a24e870c-527e-43a4-9eb7-337b49468d84" (UID: "a24e870c-527e-43a4-9eb7-337b49468d84"). InnerVolumeSpecName "kube-api-access-9k8sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:12:58.479831 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.479792    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-token\" (UniqueName: \"kubernetes.io/projected/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-token\") pod \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") "
Nov 02 02:12:58.479831 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.479837    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-workload-socket\") pod \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") "
Nov 02 02:12:58.480059 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.479916    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istiod-ca-cert\") pod \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") "
Nov 02 02:12:58.480059 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.479933    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-envoy\") pod \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") "
Nov 02 02:12:58.480059 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.480001    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-workload-certs\") pod \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") "
Nov 02 02:12:58.480059 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.480043    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-data\") pod \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") "
Nov 02 02:12:58.480209 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.480077    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-credential-socket\") pod \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") "
Nov 02 02:12:58.480209 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.480113    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dbfefd1-0270-4d23-be4b-b4f3561e877e-kube-api-access-4xdpq\" (UniqueName: \"kubernetes.io/projected/6dbfefd1-0270-4d23-be4b-b4f3561e877e-kube-api-access-4xdpq\") pod \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") "
Nov 02 02:12:58.480209 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.480173    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-podinfo\") pod \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\" (UID: \"6dbfefd1-0270-4d23-be4b-b4f3561e877e\") "
Nov 02 02:12:58.480209 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.480173    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-workload-socket" pod "6dbfefd1-0270-4d23-be4b-b4f3561e877e" (UID: "6dbfefd1-0270-4d23-be4b-b4f3561e877e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:58.480474 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.480371    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istiod-ca-cert" pod "6dbfefd1-0270-4d23-be4b-b4f3561e877e" (UID: "6dbfefd1-0270-4d23-be4b-b4f3561e877e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:12:58.480474 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.480408    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-data" pod "6dbfefd1-0270-4d23-be4b-b4f3561e877e" (UID: "6dbfefd1-0270-4d23-be4b-b4f3561e877e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:58.480474 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.480448    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-credential-socket" pod "6dbfefd1-0270-4d23-be4b-b4f3561e877e" (UID: "6dbfefd1-0270-4d23-be4b-b4f3561e877e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:58.480641 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.480605    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-workload-certs" pod "6dbfefd1-0270-4d23-be4b-b4f3561e877e" (UID: "6dbfefd1-0270-4d23-be4b-b4f3561e877e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:58.480865 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.480779    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389a930e-b478-448a-a3e6-a3466a362c22-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:58.480865 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.480796    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k8sr\" (UniqueName: \"kubernetes.io/projected/a24e870c-527e-43a4-9eb7-337b49468d84-kube-api-access-9k8sr\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:58.480865 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.480806    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389a930e-b478-448a-a3e6-a3466a362c22-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:58.480865 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.480815    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389a930e-b478-448a-a3e6-a3466a362c22-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:58.480865 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.480823    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:58.480865 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.480833    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389a930e-b478-448a-a3e6-a3466a362c22-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:58.480865 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.480841    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzc4d\" (UniqueName: \"kubernetes.io/projected/389a930e-b478-448a-a3e6-a3466a362c22-kube-api-access-jzc4d\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:58.480865 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.480851    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a24e870c-527e-43a4-9eb7-337b49468d84-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:58.480865 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.480859    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:58.480865 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.480869    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:58.481433 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.480877    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a24e870c-527e-43a4-9eb7-337b49468d84-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:58.481433 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.480886    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:58.481433 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.480894    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:58.482472 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.482445    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-token" pod "6dbfefd1-0270-4d23-be4b-b4f3561e877e" (UID: "6dbfefd1-0270-4d23-be4b-b4f3561e877e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:12:58.482472 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.482444    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-envoy" pod "6dbfefd1-0270-4d23-be4b-b4f3561e877e" (UID: "6dbfefd1-0270-4d23-be4b-b4f3561e877e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:58.482968 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.482953    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dbfefd1-0270-4d23-be4b-b4f3561e877e-kube-api-access-4xdpq" pod "6dbfefd1-0270-4d23-be4b-b4f3561e877e" (UID: "6dbfefd1-0270-4d23-be4b-b4f3561e877e"). InnerVolumeSpecName "kube-api-access-4xdpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:12:58.483036 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.482968    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-podinfo" pod "6dbfefd1-0270-4d23-be4b-b4f3561e877e" (UID: "6dbfefd1-0270-4d23-be4b-b4f3561e877e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:12:58.582653 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.582618    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:58.582653 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.582656    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xdpq\" (UniqueName: \"kubernetes.io/projected/6dbfefd1-0270-4d23-be4b-b4f3561e877e-kube-api-access-4xdpq\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:58.582941 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.582672    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:58.582941 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:58.582685    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbfefd1-0270-4d23-be4b-b4f3561e877e-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:59.148053 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.148018    5639 generic.go:356] "Generic (PLEG): container finished" podID="6dbfefd1-0270-4d23-be4b-b4f3561e877e" containerID="12a4cadebcebad0565852db61fa3df34f6944343b6b7cdaf45355402f12da42e" exitCode=0
Nov 02 02:12:59.148606 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.148079    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"
Nov 02 02:12:59.148606 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.148104    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn" event={"ID":"6dbfefd1-0270-4d23-be4b-b4f3561e877e","Type":"ContainerDied","Data":"12a4cadebcebad0565852db61fa3df34f6944343b6b7cdaf45355402f12da42e"}
Nov 02 02:12:59.148606 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.148167    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn" event={"ID":"6dbfefd1-0270-4d23-be4b-b4f3561e877e","Type":"ContainerDied","Data":"73c44d21972ff2c38e934d9400764448970a413b736d1c912a86afa2942e0810"}
Nov 02 02:12:59.148606 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.148203    5639 scope.go:122] "RemoveContainer" containerID="12a4cadebcebad0565852db61fa3df34f6944343b6b7cdaf45355402f12da42e"
Nov 02 02:12:59.152895 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.152880    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"
Nov 02 02:12:59.152992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.152908    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh" event={"ID":"389a930e-b478-448a-a3e6-a3466a362c22","Type":"ContainerDied","Data":"a5398e9b87fa7799fcda4d99d214a8841f279a81f3290680cec0a49c3df5fcfb"}
Nov 02 02:12:59.154405 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.154384    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn"
Nov 02 02:12:59.154618 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.154423    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn" event={"ID":"a24e870c-527e-43a4-9eb7-337b49468d84","Type":"ContainerDied","Data":"62287779bf4daab74140c2eb93d2363d134dcf3d01f96a89b382b6036b50fff1"}
Nov 02 02:12:59.154727 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.154646    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62287779bf4daab74140c2eb93d2363d134dcf3d01f96a89b382b6036b50fff1"
Nov 02 02:12:59.227098 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.227068    5639 scope.go:122] "RemoveContainer" containerID="12a4cadebcebad0565852db61fa3df34f6944343b6b7cdaf45355402f12da42e"
Nov 02 02:12:59.227447 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:12:59.227424    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12a4cadebcebad0565852db61fa3df34f6944343b6b7cdaf45355402f12da42e\": container with ID starting with 12a4cadebcebad0565852db61fa3df34f6944343b6b7cdaf45355402f12da42e not found: ID does not exist" containerID="12a4cadebcebad0565852db61fa3df34f6944343b6b7cdaf45355402f12da42e"
Nov 02 02:12:59.227561 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.227460    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12a4cadebcebad0565852db61fa3df34f6944343b6b7cdaf45355402f12da42e"} err="failed to get container status \"12a4cadebcebad0565852db61fa3df34f6944343b6b7cdaf45355402f12da42e\": rpc error: code = NotFound desc = could not find container \"12a4cadebcebad0565852db61fa3df34f6944343b6b7cdaf45355402f12da42e\": container with ID starting with 12a4cadebcebad0565852db61fa3df34f6944343b6b7cdaf45355402f12da42e not found: ID does not exist"
Nov 02 02:12:59.227561 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.227485    5639 scope.go:122] "RemoveContainer" containerID="b0ed4f58003e70fb22a6d62908ee68a7cc3dac080f1655cda0823518dc4e5521"
Nov 02 02:12:59.366225 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.366197    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"]
Nov 02 02:12:59.372353 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.372326    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4zcnn"]
Nov 02 02:12:59.373049 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.373031    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:12:59.481909 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.481874    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"]
Nov 02 02:12:59.487481 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.487431    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h98bh"]
Nov 02 02:12:59.495062 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.493999    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-credential-socket\") pod \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") "
Nov 02 02:12:59.495062 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.494071    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-token\" (UniqueName: \"kubernetes.io/projected/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-token\") pod \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") "
Nov 02 02:12:59.495062 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.494138    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-envoy\") pod \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") "
Nov 02 02:12:59.495062 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.494193    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-data\") pod \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") "
Nov 02 02:12:59.495062 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.494228    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-workload-socket\") pod \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") "
Nov 02 02:12:59.495062 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.494357    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istiod-ca-cert\") pod \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") "
Nov 02 02:12:59.495062 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.494393    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c89cdf8-4ee0-444d-a8a3-8371f375c432-kube-api-access-27rsm\" (UniqueName: \"kubernetes.io/projected/0c89cdf8-4ee0-444d-a8a3-8371f375c432-kube-api-access-27rsm\") pod \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") "
Nov 02 02:12:59.495062 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.494457    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-podinfo\") pod \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") "
Nov 02 02:12:59.495062 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.494495    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-workload-certs\") pod \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\" (UID: \"0c89cdf8-4ee0-444d-a8a3-8371f375c432\") "
Nov 02 02:12:59.495676 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.495615    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-workload-socket" pod "0c89cdf8-4ee0-444d-a8a3-8371f375c432" (UID: "0c89cdf8-4ee0-444d-a8a3-8371f375c432"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:59.495805 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.495780    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-workload-certs" pod "0c89cdf8-4ee0-444d-a8a3-8371f375c432" (UID: "0c89cdf8-4ee0-444d-a8a3-8371f375c432"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:59.495925 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.495896    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istiod-ca-cert" pod "0c89cdf8-4ee0-444d-a8a3-8371f375c432" (UID: "0c89cdf8-4ee0-444d-a8a3-8371f375c432"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:12:59.496142 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.496118    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-credential-socket" pod "0c89cdf8-4ee0-444d-a8a3-8371f375c432" (UID: "0c89cdf8-4ee0-444d-a8a3-8371f375c432"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:59.496459 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.496433    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-data" pod "0c89cdf8-4ee0-444d-a8a3-8371f375c432" (UID: "0c89cdf8-4ee0-444d-a8a3-8371f375c432"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:59.497839 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.497814    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-envoy" pod "0c89cdf8-4ee0-444d-a8a3-8371f375c432" (UID: "0c89cdf8-4ee0-444d-a8a3-8371f375c432"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:59.498148 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.498128    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-podinfo" pod "0c89cdf8-4ee0-444d-a8a3-8371f375c432" (UID: "0c89cdf8-4ee0-444d-a8a3-8371f375c432"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:12:59.498341 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.498312    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c89cdf8-4ee0-444d-a8a3-8371f375c432-kube-api-access-27rsm" pod "0c89cdf8-4ee0-444d-a8a3-8371f375c432" (UID: "0c89cdf8-4ee0-444d-a8a3-8371f375c432"). InnerVolumeSpecName "kube-api-access-27rsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:12:59.498413 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.498353    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-token" pod "0c89cdf8-4ee0-444d-a8a3-8371f375c432" (UID: "0c89cdf8-4ee0-444d-a8a3-8371f375c432"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:12:59.595619 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.595579    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:59.595619 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.595606    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:59.595619 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.595616    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:59.595619 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.595625    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:59.595898 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.595635    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27rsm\" (UniqueName: \"kubernetes.io/projected/0c89cdf8-4ee0-444d-a8a3-8371f375c432-kube-api-access-27rsm\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:59.595898 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.595644    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:59.595898 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.595653    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:59.595898 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.595662    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c89cdf8-4ee0-444d-a8a3-8371f375c432-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:59.595898 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:12:59.595669    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c89cdf8-4ee0-444d-a8a3-8371f375c432-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:00.164368 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:00.164336    5639 generic.go:356] "Generic (PLEG): container finished" podID="0c89cdf8-4ee0-444d-a8a3-8371f375c432" containerID="d7b9e5030228b7b7a01446d7629759ee09f38d1df8df035055f4ee3603f2101d" exitCode=0
Nov 02 02:13:00.164877 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:00.164406    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"
Nov 02 02:13:00.164877 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:00.164418    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5" event={"ID":"0c89cdf8-4ee0-444d-a8a3-8371f375c432","Type":"ContainerDied","Data":"d7b9e5030228b7b7a01446d7629759ee09f38d1df8df035055f4ee3603f2101d"}
Nov 02 02:13:00.164877 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:00.164519    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5" event={"ID":"0c89cdf8-4ee0-444d-a8a3-8371f375c432","Type":"ContainerDied","Data":"3c10bf8af21e02577118f70675b24bc143cb1e09ee45798db34124bdc1b84981"}
Nov 02 02:13:00.164877 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:00.164553    5639 scope.go:122] "RemoveContainer" containerID="d7b9e5030228b7b7a01446d7629759ee09f38d1df8df035055f4ee3603f2101d"
Nov 02 02:13:00.175588 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:00.175572    5639 scope.go:122] "RemoveContainer" containerID="5f1ba773868853577afca0f3215588dbd1a7d3afe9196e4c74e77b36ddd6b058"
Nov 02 02:13:00.185149 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:00.185126    5639 scope.go:122] "RemoveContainer" containerID="d7b9e5030228b7b7a01446d7629759ee09f38d1df8df035055f4ee3603f2101d"
Nov 02 02:13:00.185442 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:13:00.185421    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7b9e5030228b7b7a01446d7629759ee09f38d1df8df035055f4ee3603f2101d\": container with ID starting with d7b9e5030228b7b7a01446d7629759ee09f38d1df8df035055f4ee3603f2101d not found: ID does not exist" containerID="d7b9e5030228b7b7a01446d7629759ee09f38d1df8df035055f4ee3603f2101d"
Nov 02 02:13:00.185491 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:00.185453    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b9e5030228b7b7a01446d7629759ee09f38d1df8df035055f4ee3603f2101d"} err="failed to get container status \"d7b9e5030228b7b7a01446d7629759ee09f38d1df8df035055f4ee3603f2101d\": rpc error: code = NotFound desc = could not find container \"d7b9e5030228b7b7a01446d7629759ee09f38d1df8df035055f4ee3603f2101d\": container with ID starting with d7b9e5030228b7b7a01446d7629759ee09f38d1df8df035055f4ee3603f2101d not found: ID does not exist"
Nov 02 02:13:00.185491 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:00.185472    5639 scope.go:122] "RemoveContainer" containerID="5f1ba773868853577afca0f3215588dbd1a7d3afe9196e4c74e77b36ddd6b058"
Nov 02 02:13:00.185706 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:13:00.185691    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f1ba773868853577afca0f3215588dbd1a7d3afe9196e4c74e77b36ddd6b058\": container with ID starting with 5f1ba773868853577afca0f3215588dbd1a7d3afe9196e4c74e77b36ddd6b058 not found: ID does not exist" containerID="5f1ba773868853577afca0f3215588dbd1a7d3afe9196e4c74e77b36ddd6b058"
Nov 02 02:13:00.185749 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:00.185709    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f1ba773868853577afca0f3215588dbd1a7d3afe9196e4c74e77b36ddd6b058"} err="failed to get container status \"5f1ba773868853577afca0f3215588dbd1a7d3afe9196e4c74e77b36ddd6b058\": rpc error: code = NotFound desc = could not find container \"5f1ba773868853577afca0f3215588dbd1a7d3afe9196e4c74e77b36ddd6b058\": container with ID starting with 5f1ba773868853577afca0f3215588dbd1a7d3afe9196e4c74e77b36ddd6b058 not found: ID does not exist"
Nov 02 02:13:00.330780 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:00.330746    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"]
Nov 02 02:13:00.335397 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:00.335368    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d0fdb9-openshift-default-5fbd9b857c-8w6m5"]
Nov 02 02:13:01.218625 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:01.218595    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c89cdf8-4ee0-444d-a8a3-8371f375c432" path="/var/lib/kubelet/pods/0c89cdf8-4ee0-444d-a8a3-8371f375c432/volumes"
Nov 02 02:13:01.219058 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:01.219044    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="389a930e-b478-448a-a3e6-a3466a362c22" path="/var/lib/kubelet/pods/389a930e-b478-448a-a3e6-a3466a362c22/volumes"
Nov 02 02:13:01.219466 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:01.219451    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dbfefd1-0270-4d23-be4b-b4f3561e877e" path="/var/lib/kubelet/pods/6dbfefd1-0270-4d23-be4b-b4f3561e877e/volumes"
Nov 02 02:13:06.480676 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:06.480641    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55"]
Nov 02 02:13:11.481567 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:11.481502    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55" podUID="c225c247-6591-43f8-b69f-d7c443f636be" containerName="payload-processing" containerID="cri-o://ab4331e0d318fccf2a6a2e82cc7f5b5cbf4d6def3f28e42588cdcc08a4d68296" gracePeriod=25
Nov 02 02:13:11.728628 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:11.728601    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55"
Nov 02 02:13:11.824179 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:11.824087    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c225c247-6591-43f8-b69f-d7c443f636be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c225c247-6591-43f8-b69f-d7c443f636be-plugins-config-volume\") pod \"c225c247-6591-43f8-b69f-d7c443f636be\" (UID: \"c225c247-6591-43f8-b69f-d7c443f636be\") "
Nov 02 02:13:11.824400 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:11.824197    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c225c247-6591-43f8-b69f-d7c443f636be-kube-api-access-zpwsj\" (UniqueName: \"kubernetes.io/projected/c225c247-6591-43f8-b69f-d7c443f636be-kube-api-access-zpwsj\") pod \"c225c247-6591-43f8-b69f-d7c443f636be\" (UID: \"c225c247-6591-43f8-b69f-d7c443f636be\") "
Nov 02 02:13:11.824563 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:11.824538    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c225c247-6591-43f8-b69f-d7c443f636be-plugins-config-volume" pod "c225c247-6591-43f8-b69f-d7c443f636be" (UID: "c225c247-6591-43f8-b69f-d7c443f636be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:13:11.824704 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:11.824687    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c225c247-6591-43f8-b69f-d7c443f636be-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:11.826514 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:11.826490    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c225c247-6591-43f8-b69f-d7c443f636be-kube-api-access-zpwsj" pod "c225c247-6591-43f8-b69f-d7c443f636be" (UID: "c225c247-6591-43f8-b69f-d7c443f636be"). InnerVolumeSpecName "kube-api-access-zpwsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:13:11.926158 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:11.926125    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpwsj\" (UniqueName: \"kubernetes.io/projected/c225c247-6591-43f8-b69f-d7c443f636be-kube-api-access-zpwsj\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:12.247642 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:12.247608    5639 generic.go:356] "Generic (PLEG): container finished" podID="c225c247-6591-43f8-b69f-d7c443f636be" containerID="ab4331e0d318fccf2a6a2e82cc7f5b5cbf4d6def3f28e42588cdcc08a4d68296" exitCode=0
Nov 02 02:13:12.247827 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:12.247650    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55"
Nov 02 02:13:12.247827 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:12.247654    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55" event={"ID":"c225c247-6591-43f8-b69f-d7c443f636be","Type":"ContainerDied","Data":"ab4331e0d318fccf2a6a2e82cc7f5b5cbf4d6def3f28e42588cdcc08a4d68296"}
Nov 02 02:13:12.247827 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:12.247696    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55" event={"ID":"c225c247-6591-43f8-b69f-d7c443f636be","Type":"ContainerDied","Data":"ea7932d8da8b688cbca3b201e1eb4a0b28678b9293bd70ce1c78bd8cf6a0d7ff"}
Nov 02 02:13:12.247827 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:12.247716    5639 scope.go:122] "RemoveContainer" containerID="ab4331e0d318fccf2a6a2e82cc7f5b5cbf4d6def3f28e42588cdcc08a4d68296"
Nov 02 02:13:12.258127 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:12.258109    5639 scope.go:122] "RemoveContainer" containerID="ab4331e0d318fccf2a6a2e82cc7f5b5cbf4d6def3f28e42588cdcc08a4d68296"
Nov 02 02:13:12.258459 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:13:12.258438    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab4331e0d318fccf2a6a2e82cc7f5b5cbf4d6def3f28e42588cdcc08a4d68296\": container with ID starting with ab4331e0d318fccf2a6a2e82cc7f5b5cbf4d6def3f28e42588cdcc08a4d68296 not found: ID does not exist" containerID="ab4331e0d318fccf2a6a2e82cc7f5b5cbf4d6def3f28e42588cdcc08a4d68296"
Nov 02 02:13:12.258544 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:12.258469    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab4331e0d318fccf2a6a2e82cc7f5b5cbf4d6def3f28e42588cdcc08a4d68296"} err="failed to get container status \"ab4331e0d318fccf2a6a2e82cc7f5b5cbf4d6def3f28e42588cdcc08a4d68296\": rpc error: code = NotFound desc = could not find container \"ab4331e0d318fccf2a6a2e82cc7f5b5cbf4d6def3f28e42588cdcc08a4d68296\": container with ID starting with ab4331e0d318fccf2a6a2e82cc7f5b5cbf4d6def3f28e42588cdcc08a4d68296 not found: ID does not exist"
Nov 02 02:13:12.411921 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:12.411883    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55"]
Nov 02 02:13:12.461340 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:12.461307    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-361fcc-856f6fb475-t5f55"]
Nov 02 02:13:12.756662 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:12.756621    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn"]
Nov 02 02:13:12.759599 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:12.759574    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-361fcc-9pvkn"]
Nov 02 02:13:13.218676 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:13.218642    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a24e870c-527e-43a4-9eb7-337b49468d84" path="/var/lib/kubelet/pods/a24e870c-527e-43a4-9eb7-337b49468d84/volumes"
Nov 02 02:13:13.219059 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:13.219045    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c225c247-6591-43f8-b69f-d7c443f636be" path="/var/lib/kubelet/pods/c225c247-6591-43f8-b69f-d7c443f636be/volumes"
Nov 02 02:13:16.501672 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:16.501642    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:13:19.572633 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:19.572595    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"]
Nov 02 02:13:19.573041 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:19.572842    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g" podUID="2e47ced9-b31c-4196-8b5c-0e980cf9297e" containerName="istio-proxy" containerID="cri-o://20a654a78b08fb51593eeeedee995b4b0fda68ad9d06afd08fb0e693ed2bacf1" gracePeriod=30
Nov 02 02:13:20.442448 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.442410    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh"]
Nov 02 02:13:20.442893 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.442880    5639 cpu_manager.go:414] "containerMap: removing container" podUID="c225c247-6591-43f8-b69f-d7c443f636be" containerName="payload-processing"
Nov 02 02:13:20.442954 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.442894    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c225c247-6591-43f8-b69f-d7c443f636be" containerName="payload-processing"
Nov 02 02:13:20.442954 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.442913    5639 cpu_manager.go:414] "containerMap: removing container" podUID="389a930e-b478-448a-a3e6-a3466a362c22" containerName="istio-proxy"
Nov 02 02:13:20.442954 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.442920    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="389a930e-b478-448a-a3e6-a3466a362c22" containerName="istio-proxy"
Nov 02 02:13:20.442954 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.442941    5639 cpu_manager.go:414] "containerMap: removing container" podUID="0c89cdf8-4ee0-444d-a8a3-8371f375c432" containerName="istio-proxy"
Nov 02 02:13:20.442954 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.442946    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c89cdf8-4ee0-444d-a8a3-8371f375c432" containerName="istio-proxy"
Nov 02 02:13:20.443123 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.442956    5639 cpu_manager.go:414] "containerMap: removing container" podUID="0c89cdf8-4ee0-444d-a8a3-8371f375c432" containerName="istio-proxy"
Nov 02 02:13:20.443123 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.442960    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c89cdf8-4ee0-444d-a8a3-8371f375c432" containerName="istio-proxy"
Nov 02 02:13:20.443123 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.442977    5639 cpu_manager.go:414] "containerMap: removing container" podUID="6dbfefd1-0270-4d23-be4b-b4f3561e877e" containerName="istio-proxy"
Nov 02 02:13:20.443123 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.442981    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dbfefd1-0270-4d23-be4b-b4f3561e877e" containerName="istio-proxy"
Nov 02 02:13:20.443123 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.442989    5639 cpu_manager.go:414] "containerMap: removing container" podUID="a24e870c-527e-43a4-9eb7-337b49468d84" containerName="revoke-keys"
Nov 02 02:13:20.443123 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.442994    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a24e870c-527e-43a4-9eb7-337b49468d84" containerName="revoke-keys"
Nov 02 02:13:20.443123 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.443054    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c89cdf8-4ee0-444d-a8a3-8371f375c432" containerName="istio-proxy"
Nov 02 02:13:20.443123 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.443061    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c89cdf8-4ee0-444d-a8a3-8371f375c432" containerName="istio-proxy"
Nov 02 02:13:20.443123 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.443072    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dbfefd1-0270-4d23-be4b-b4f3561e877e" containerName="istio-proxy"
Nov 02 02:13:20.443123 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.443078    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="a24e870c-527e-43a4-9eb7-337b49468d84" containerName="revoke-keys"
Nov 02 02:13:20.443123 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.443085    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="c225c247-6591-43f8-b69f-d7c443f636be" containerName="payload-processing"
Nov 02 02:13:20.443123 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.443091    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="389a930e-b478-448a-a3e6-a3466a362c22" containerName="istio-proxy"
Nov 02 02:13:20.445782 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.445765    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh"
Nov 02 02:13:20.503536 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.503504    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh"]
Nov 02 02:13:20.550170 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.550134    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:13:20.550389 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.550203    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:13:20.550389 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.550239    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:13:20.550485 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.550385    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:13:20.619756 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.619721    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4f337c9-0884-424c-934d-0a1015586045-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh\" (UID: \"b4f337c9-0884-424c-934d-0a1015586045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh"
Nov 02 02:13:20.620137 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.619841    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq4ml\" (UniqueName: \"kubernetes.io/projected/b4f337c9-0884-424c-934d-0a1015586045-kube-api-access-vq4ml\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh\" (UID: \"b4f337c9-0884-424c-934d-0a1015586045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh"
Nov 02 02:13:20.620137 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.619915    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4f337c9-0884-424c-934d-0a1015586045-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh\" (UID: \"b4f337c9-0884-424c-934d-0a1015586045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh"
Nov 02 02:13:20.721500 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.721422    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq4ml\" (UniqueName: \"kubernetes.io/projected/b4f337c9-0884-424c-934d-0a1015586045-kube-api-access-vq4ml\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh\" (UID: \"b4f337c9-0884-424c-934d-0a1015586045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh"
Nov 02 02:13:20.721500 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.721490    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4f337c9-0884-424c-934d-0a1015586045-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh\" (UID: \"b4f337c9-0884-424c-934d-0a1015586045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh"
Nov 02 02:13:20.721694 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.721531    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4f337c9-0884-424c-934d-0a1015586045-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh\" (UID: \"b4f337c9-0884-424c-934d-0a1015586045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh"
Nov 02 02:13:20.722355 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.722322    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4f337c9-0884-424c-934d-0a1015586045-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh\" (UID: \"b4f337c9-0884-424c-934d-0a1015586045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh"
Nov 02 02:13:20.724161 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.724139    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4f337c9-0884-424c-934d-0a1015586045-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh\" (UID: \"b4f337c9-0884-424c-934d-0a1015586045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh"
Nov 02 02:13:20.783979 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:20.783943    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq4ml\" (UniqueName: \"kubernetes.io/projected/b4f337c9-0884-424c-934d-0a1015586045-kube-api-access-vq4ml\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh\" (UID: \"b4f337c9-0884-424c-934d-0a1015586045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh"
Nov 02 02:13:21.057613 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:21.057527    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh"
Nov 02 02:13:21.428796 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:21.428764    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh"]
Nov 02 02:13:21.431056 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:13:21.431025    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f337c9_0884_424c_934d_0a1015586045.slice/crio-51da6ff09b4486314a14a2dd8a75836436566a760a9ef0e7e2169c72cd7405ae WatchSource:0}: Error finding container 51da6ff09b4486314a14a2dd8a75836436566a760a9ef0e7e2169c72cd7405ae: Status 404 returned error can't find the container with id 51da6ff09b4486314a14a2dd8a75836436566a760a9ef0e7e2169c72cd7405ae
Nov 02 02:13:22.315142 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:22.315095    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh" event={"ID":"b4f337c9-0884-424c-934d-0a1015586045","Type":"ContainerStarted","Data":"4e363bde5862cf6f5bddda79a1cba82802ed69a76b8d0b78dd92731519bdfd69"}
Nov 02 02:13:22.315670 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:22.315200    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh" event={"ID":"b4f337c9-0884-424c-934d-0a1015586045","Type":"ContainerStarted","Data":"51da6ff09b4486314a14a2dd8a75836436566a760a9ef0e7e2169c72cd7405ae"}
Nov 02 02:13:22.431169 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:22.431120    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh" podStartSLOduration=2.4311063969999998 podStartE2EDuration="2.431106397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:13:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:13:22.429843774 +0000 UTC m=+2867.333525889" watchObservedRunningTime="2026-11-02 02:13:22.431106397 +0000 UTC m=+2867.334788511"
Nov 02 02:13:23.322649 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:23.322613    5639 generic.go:356] "Generic (PLEG): container finished" podID="b4f337c9-0884-424c-934d-0a1015586045" containerID="4e363bde5862cf6f5bddda79a1cba82802ed69a76b8d0b78dd92731519bdfd69" exitCode=0
Nov 02 02:13:23.323064 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:23.322703    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh" event={"ID":"b4f337c9-0884-424c-934d-0a1015586045","Type":"ContainerDied","Data":"4e363bde5862cf6f5bddda79a1cba82802ed69a76b8d0b78dd92731519bdfd69"}
Nov 02 02:13:24.468434 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.468408    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh"
Nov 02 02:13:24.558058 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.558027    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4f337c9-0884-424c-934d-0a1015586045-kube-api-access-vq4ml\" (UniqueName: \"kubernetes.io/projected/b4f337c9-0884-424c-934d-0a1015586045-kube-api-access-vq4ml\") pod \"b4f337c9-0884-424c-934d-0a1015586045\" (UID: \"b4f337c9-0884-424c-934d-0a1015586045\") "
Nov 02 02:13:24.558239 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.558106    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4f337c9-0884-424c-934d-0a1015586045-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4f337c9-0884-424c-934d-0a1015586045-maas-api-xks-ca\") pod \"b4f337c9-0884-424c-934d-0a1015586045\" (UID: \"b4f337c9-0884-424c-934d-0a1015586045\") "
Nov 02 02:13:24.558339 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.558271    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4f337c9-0884-424c-934d-0a1015586045-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4f337c9-0884-424c-934d-0a1015586045-maas-api-openshift-ca\") pod \"b4f337c9-0884-424c-934d-0a1015586045\" (UID: \"b4f337c9-0884-424c-934d-0a1015586045\") "
Nov 02 02:13:24.558627 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.558598    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f337c9-0884-424c-934d-0a1015586045-maas-api-openshift-ca" pod "b4f337c9-0884-424c-934d-0a1015586045" (UID: "b4f337c9-0884-424c-934d-0a1015586045"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:13:24.560491 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.560468    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4f337c9-0884-424c-934d-0a1015586045-maas-api-xks-ca" pod "b4f337c9-0884-424c-934d-0a1015586045" (UID: "b4f337c9-0884-424c-934d-0a1015586045"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:13:24.560491 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.560475    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f337c9-0884-424c-934d-0a1015586045-kube-api-access-vq4ml" pod "b4f337c9-0884-424c-934d-0a1015586045" (UID: "b4f337c9-0884-424c-934d-0a1015586045"). InnerVolumeSpecName "kube-api-access-vq4ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:13:24.659390 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.659358    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq4ml\" (UniqueName: \"kubernetes.io/projected/b4f337c9-0884-424c-934d-0a1015586045-kube-api-access-vq4ml\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:24.659390 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.659390    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4f337c9-0884-424c-934d-0a1015586045-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:24.659590 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.659400    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4f337c9-0884-424c-934d-0a1015586045-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:24.822109 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.822086    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:13:24.961856 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.961764    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-podinfo\") pod \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") "
Nov 02 02:13:24.961856 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.961817    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-token\" (UniqueName: \"kubernetes.io/projected/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-token\") pod \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") "
Nov 02 02:13:24.962100 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.961859    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-envoy\") pod \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") "
Nov 02 02:13:24.962100 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.961950    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-workload-socket\") pod \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") "
Nov 02 02:13:24.962100 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.962007    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-credential-socket\") pod \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") "
Nov 02 02:13:24.962100 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.962035    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-workload-certs\") pod \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") "
Nov 02 02:13:24.962330 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.962124    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istiod-ca-cert\") pod \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") "
Nov 02 02:13:24.962330 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.962167    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e47ced9-b31c-4196-8b5c-0e980cf9297e-kube-api-access-2tb5x\" (UniqueName: \"kubernetes.io/projected/2e47ced9-b31c-4196-8b5c-0e980cf9297e-kube-api-access-2tb5x\") pod \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") "
Nov 02 02:13:24.962330 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.962201    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-data\") pod \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\" (UID: \"2e47ced9-b31c-4196-8b5c-0e980cf9297e\") "
Nov 02 02:13:24.962330 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.962262    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-workload-socket" pod "2e47ced9-b31c-4196-8b5c-0e980cf9297e" (UID: "2e47ced9-b31c-4196-8b5c-0e980cf9297e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:13:24.962557 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.962489    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-workload-certs" pod "2e47ced9-b31c-4196-8b5c-0e980cf9297e" (UID: "2e47ced9-b31c-4196-8b5c-0e980cf9297e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:13:24.962557 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.962502    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-credential-socket" pod "2e47ced9-b31c-4196-8b5c-0e980cf9297e" (UID: "2e47ced9-b31c-4196-8b5c-0e980cf9297e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:13:24.962663 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.962634    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:24.962663 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.962654    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:24.962769 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.962668    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:24.962829 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.962780    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istiod-ca-cert" pod "2e47ced9-b31c-4196-8b5c-0e980cf9297e" (UID: "2e47ced9-b31c-4196-8b5c-0e980cf9297e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:13:24.962894 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.962880    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-data" pod "2e47ced9-b31c-4196-8b5c-0e980cf9297e" (UID: "2e47ced9-b31c-4196-8b5c-0e980cf9297e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:13:24.964624 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.964593    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-envoy" pod "2e47ced9-b31c-4196-8b5c-0e980cf9297e" (UID: "2e47ced9-b31c-4196-8b5c-0e980cf9297e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:13:24.964733 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.964682    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e47ced9-b31c-4196-8b5c-0e980cf9297e-kube-api-access-2tb5x" pod "2e47ced9-b31c-4196-8b5c-0e980cf9297e" (UID: "2e47ced9-b31c-4196-8b5c-0e980cf9297e"). InnerVolumeSpecName "kube-api-access-2tb5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:13:24.964898 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.964877    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-podinfo" pod "2e47ced9-b31c-4196-8b5c-0e980cf9297e" (UID: "2e47ced9-b31c-4196-8b5c-0e980cf9297e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:13:24.964944 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:24.964885    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-token" pod "2e47ced9-b31c-4196-8b5c-0e980cf9297e" (UID: "2e47ced9-b31c-4196-8b5c-0e980cf9297e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:13:25.063847 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:25.063809    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:25.063847 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:25.063840    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tb5x\" (UniqueName: \"kubernetes.io/projected/2e47ced9-b31c-4196-8b5c-0e980cf9297e-kube-api-access-2tb5x\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:25.063847 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:25.063852    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:25.064061 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:25.063860    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:25.064061 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:25.063868    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:25.064061 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:25.063877    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e47ced9-b31c-4196-8b5c-0e980cf9297e-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:25.336241 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:25.336208    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh"
Nov 02 02:13:25.336426 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:25.336207    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh" event={"ID":"b4f337c9-0884-424c-934d-0a1015586045","Type":"ContainerDied","Data":"51da6ff09b4486314a14a2dd8a75836436566a760a9ef0e7e2169c72cd7405ae"}
Nov 02 02:13:25.336426 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:25.336355    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51da6ff09b4486314a14a2dd8a75836436566a760a9ef0e7e2169c72cd7405ae"
Nov 02 02:13:25.338224 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:25.338195    5639 generic.go:356] "Generic (PLEG): container finished" podID="2e47ced9-b31c-4196-8b5c-0e980cf9297e" containerID="20a654a78b08fb51593eeeedee995b4b0fda68ad9d06afd08fb0e693ed2bacf1" exitCode=0
Nov 02 02:13:25.338476 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:25.338241    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"
Nov 02 02:13:25.338476 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:25.338296    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g" event={"ID":"2e47ced9-b31c-4196-8b5c-0e980cf9297e","Type":"ContainerDied","Data":"20a654a78b08fb51593eeeedee995b4b0fda68ad9d06afd08fb0e693ed2bacf1"}
Nov 02 02:13:25.338476 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:25.338360    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g" event={"ID":"2e47ced9-b31c-4196-8b5c-0e980cf9297e","Type":"ContainerDied","Data":"cf259a5e3586db09d56518a847f0dc100342d52734f814adad1919c68e2c3b51"}
Nov 02 02:13:25.338476 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:25.338410    5639 scope.go:122] "RemoveContainer" containerID="20a654a78b08fb51593eeeedee995b4b0fda68ad9d06afd08fb0e693ed2bacf1"
Nov 02 02:13:25.348255 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:25.348233    5639 scope.go:122] "RemoveContainer" containerID="8e11838eba77646636a18a852d2371239793b4c0c738887baa3cd07a7b902250"
Nov 02 02:13:25.356847 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:25.356814    5639 scope.go:122] "RemoveContainer" containerID="20a654a78b08fb51593eeeedee995b4b0fda68ad9d06afd08fb0e693ed2bacf1"
Nov 02 02:13:25.357117 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:13:25.357096    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20a654a78b08fb51593eeeedee995b4b0fda68ad9d06afd08fb0e693ed2bacf1\": container with ID starting with 20a654a78b08fb51593eeeedee995b4b0fda68ad9d06afd08fb0e693ed2bacf1 not found: ID does not exist" containerID="20a654a78b08fb51593eeeedee995b4b0fda68ad9d06afd08fb0e693ed2bacf1"
Nov 02 02:13:25.357168 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:25.357124    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20a654a78b08fb51593eeeedee995b4b0fda68ad9d06afd08fb0e693ed2bacf1"} err="failed to get container status \"20a654a78b08fb51593eeeedee995b4b0fda68ad9d06afd08fb0e693ed2bacf1\": rpc error: code = NotFound desc = could not find container \"20a654a78b08fb51593eeeedee995b4b0fda68ad9d06afd08fb0e693ed2bacf1\": container with ID starting with 20a654a78b08fb51593eeeedee995b4b0fda68ad9d06afd08fb0e693ed2bacf1 not found: ID does not exist"
Nov 02 02:13:25.357168 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:25.357142    5639 scope.go:122] "RemoveContainer" containerID="8e11838eba77646636a18a852d2371239793b4c0c738887baa3cd07a7b902250"
Nov 02 02:13:25.357393 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:13:25.357374    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e11838eba77646636a18a852d2371239793b4c0c738887baa3cd07a7b902250\": container with ID starting with 8e11838eba77646636a18a852d2371239793b4c0c738887baa3cd07a7b902250 not found: ID does not exist" containerID="8e11838eba77646636a18a852d2371239793b4c0c738887baa3cd07a7b902250"
Nov 02 02:13:25.357443 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:25.357401    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e11838eba77646636a18a852d2371239793b4c0c738887baa3cd07a7b902250"} err="failed to get container status \"8e11838eba77646636a18a852d2371239793b4c0c738887baa3cd07a7b902250\": rpc error: code = NotFound desc = could not find container \"8e11838eba77646636a18a852d2371239793b4c0c738887baa3cd07a7b902250\": container with ID starting with 8e11838eba77646636a18a852d2371239793b4c0c738887baa3cd07a7b902250 not found: ID does not exist"
Nov 02 02:13:25.502057 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:25.501982    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"]
Nov 02 02:13:25.505470 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:25.505444    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-361fcc-openshift-default-8cdb659b5-x9x7g"]
Nov 02 02:13:27.219769 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:27.219733    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e47ced9-b31c-4196-8b5c-0e980cf9297e" path="/var/lib/kubelet/pods/2e47ced9-b31c-4196-8b5c-0e980cf9297e/volumes"
Nov 02 02:13:27.419873 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:13:27.419836    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:13:31.734762 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:31.734723    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg"]
Nov 02 02:13:31.840341 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:31.840305    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8"]
Nov 02 02:13:36.547773 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:36.547731    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh"]
Nov 02 02:13:36.552937 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:36.552901    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-257466-vhvgh"]
Nov 02 02:13:36.735575 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:36.735527    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg" podUID="d8e14587-d00a-43a2-8eba-b907a34a3d91" containerName="payload-processing" containerID="cri-o://3c664d046bbc02d4eba48cff4905bff72dec56f707646ccb1a4e928bc6afd8af" gracePeriod=25
Nov 02 02:13:36.840617 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:36.840542    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8" podUID="f35a24df-cd1c-4768-9596-3ef625d19864" containerName="payload-pre-processing" containerID="cri-o://c096363a247a4161af13da34b3a653a74e8bd26e51df3a6cfd8e92245acc0a80" gracePeriod=25
Nov 02 02:13:36.870133 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:13:36.870102    5639 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35a24df_cd1c_4768_9596_3ef625d19864.slice/crio-c096363a247a4161af13da34b3a653a74e8bd26e51df3a6cfd8e92245acc0a80.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35a24df_cd1c_4768_9596_3ef625d19864.slice/crio-conmon-c096363a247a4161af13da34b3a653a74e8bd26e51df3a6cfd8e92245acc0a80.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 02:13:37.001576 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.001546    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg"
Nov 02 02:13:37.082887 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.082861    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8e14587-d00a-43a2-8eba-b907a34a3d91-kube-api-access-6qsqh\" (UniqueName: \"kubernetes.io/projected/d8e14587-d00a-43a2-8eba-b907a34a3d91-kube-api-access-6qsqh\") pod \"d8e14587-d00a-43a2-8eba-b907a34a3d91\" (UID: \"d8e14587-d00a-43a2-8eba-b907a34a3d91\") "
Nov 02 02:13:37.083043 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.082920    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8e14587-d00a-43a2-8eba-b907a34a3d91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e14587-d00a-43a2-8eba-b907a34a3d91-plugins-config-volume\") pod \"d8e14587-d00a-43a2-8eba-b907a34a3d91\" (UID: \"d8e14587-d00a-43a2-8eba-b907a34a3d91\") "
Nov 02 02:13:37.083320 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.083292    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8e14587-d00a-43a2-8eba-b907a34a3d91-plugins-config-volume" pod "d8e14587-d00a-43a2-8eba-b907a34a3d91" (UID: "d8e14587-d00a-43a2-8eba-b907a34a3d91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:13:37.085150 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.085124    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e14587-d00a-43a2-8eba-b907a34a3d91-kube-api-access-6qsqh" pod "d8e14587-d00a-43a2-8eba-b907a34a3d91" (UID: "d8e14587-d00a-43a2-8eba-b907a34a3d91"). InnerVolumeSpecName "kube-api-access-6qsqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:13:37.096886 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.096828    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8"
Nov 02 02:13:37.184136 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.184097    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f35a24df-cd1c-4768-9596-3ef625d19864-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f35a24df-cd1c-4768-9596-3ef625d19864-plugins-config-volume\") pod \"f35a24df-cd1c-4768-9596-3ef625d19864\" (UID: \"f35a24df-cd1c-4768-9596-3ef625d19864\") "
Nov 02 02:13:37.184394 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.184154    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f35a24df-cd1c-4768-9596-3ef625d19864-kube-api-access-zx57b\" (UniqueName: \"kubernetes.io/projected/f35a24df-cd1c-4768-9596-3ef625d19864-kube-api-access-zx57b\") pod \"f35a24df-cd1c-4768-9596-3ef625d19864\" (UID: \"f35a24df-cd1c-4768-9596-3ef625d19864\") "
Nov 02 02:13:37.184539 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.184513    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f35a24df-cd1c-4768-9596-3ef625d19864-plugins-config-volume" pod "f35a24df-cd1c-4768-9596-3ef625d19864" (UID: "f35a24df-cd1c-4768-9596-3ef625d19864"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:13:37.184627 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.184614    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e14587-d00a-43a2-8eba-b907a34a3d91-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:37.184669 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.184630    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f35a24df-cd1c-4768-9596-3ef625d19864-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:37.184669 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.184641    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qsqh\" (UniqueName: \"kubernetes.io/projected/d8e14587-d00a-43a2-8eba-b907a34a3d91-kube-api-access-6qsqh\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:37.186472 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.186450    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f35a24df-cd1c-4768-9596-3ef625d19864-kube-api-access-zx57b" pod "f35a24df-cd1c-4768-9596-3ef625d19864" (UID: "f35a24df-cd1c-4768-9596-3ef625d19864"). InnerVolumeSpecName "kube-api-access-zx57b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:13:37.218016 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.217985    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4f337c9-0884-424c-934d-0a1015586045" path="/var/lib/kubelet/pods/b4f337c9-0884-424c-934d-0a1015586045/volumes"
Nov 02 02:13:37.285359 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.285329    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx57b\" (UniqueName: \"kubernetes.io/projected/f35a24df-cd1c-4768-9596-3ef625d19864-kube-api-access-zx57b\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:37.383875 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.383787    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:13:37.410913 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.410883    5639 generic.go:356] "Generic (PLEG): container finished" podID="f35a24df-cd1c-4768-9596-3ef625d19864" containerID="c096363a247a4161af13da34b3a653a74e8bd26e51df3a6cfd8e92245acc0a80" exitCode=0
Nov 02 02:13:37.411103 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.410931    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8"
Nov 02 02:13:37.411103 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.410957    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8" event={"ID":"f35a24df-cd1c-4768-9596-3ef625d19864","Type":"ContainerDied","Data":"c096363a247a4161af13da34b3a653a74e8bd26e51df3a6cfd8e92245acc0a80"}
Nov 02 02:13:37.411103 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.411001    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8" event={"ID":"f35a24df-cd1c-4768-9596-3ef625d19864","Type":"ContainerDied","Data":"af1400295c6ca94ed96f0a19221c7d61a8a65ff97971011775f40ed325f65129"}
Nov 02 02:13:37.411103 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.411032    5639 scope.go:122] "RemoveContainer" containerID="c096363a247a4161af13da34b3a653a74e8bd26e51df3a6cfd8e92245acc0a80"
Nov 02 02:13:37.412559 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.412530    5639 generic.go:356] "Generic (PLEG): container finished" podID="d8e14587-d00a-43a2-8eba-b907a34a3d91" containerID="3c664d046bbc02d4eba48cff4905bff72dec56f707646ccb1a4e928bc6afd8af" exitCode=0
Nov 02 02:13:37.412712 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.412572    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg"
Nov 02 02:13:37.412759 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.412614    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg" event={"ID":"d8e14587-d00a-43a2-8eba-b907a34a3d91","Type":"ContainerDied","Data":"3c664d046bbc02d4eba48cff4905bff72dec56f707646ccb1a4e928bc6afd8af"}
Nov 02 02:13:37.412821 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.412781    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg" event={"ID":"d8e14587-d00a-43a2-8eba-b907a34a3d91","Type":"ContainerDied","Data":"42b9c95fa87ae16d4b6c00ae190ded3bc6e7ea896178c86a084c2f6bb5e2d689"}
Nov 02 02:13:37.420945 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.420928    5639 scope.go:122] "RemoveContainer" containerID="c096363a247a4161af13da34b3a653a74e8bd26e51df3a6cfd8e92245acc0a80"
Nov 02 02:13:37.421248 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:13:37.421225    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c096363a247a4161af13da34b3a653a74e8bd26e51df3a6cfd8e92245acc0a80\": container with ID starting with c096363a247a4161af13da34b3a653a74e8bd26e51df3a6cfd8e92245acc0a80 not found: ID does not exist" containerID="c096363a247a4161af13da34b3a653a74e8bd26e51df3a6cfd8e92245acc0a80"
Nov 02 02:13:37.421357 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.421256    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c096363a247a4161af13da34b3a653a74e8bd26e51df3a6cfd8e92245acc0a80"} err="failed to get container status \"c096363a247a4161af13da34b3a653a74e8bd26e51df3a6cfd8e92245acc0a80\": rpc error: code = NotFound desc = could not find container \"c096363a247a4161af13da34b3a653a74e8bd26e51df3a6cfd8e92245acc0a80\": container with ID starting with c096363a247a4161af13da34b3a653a74e8bd26e51df3a6cfd8e92245acc0a80 not found: ID does not exist"
Nov 02 02:13:37.421357 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.421274    5639 scope.go:122] "RemoveContainer" containerID="3c664d046bbc02d4eba48cff4905bff72dec56f707646ccb1a4e928bc6afd8af"
Nov 02 02:13:37.430150 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.430115    5639 scope.go:122] "RemoveContainer" containerID="3c664d046bbc02d4eba48cff4905bff72dec56f707646ccb1a4e928bc6afd8af"
Nov 02 02:13:37.430493 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:13:37.430472    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c664d046bbc02d4eba48cff4905bff72dec56f707646ccb1a4e928bc6afd8af\": container with ID starting with 3c664d046bbc02d4eba48cff4905bff72dec56f707646ccb1a4e928bc6afd8af not found: ID does not exist" containerID="3c664d046bbc02d4eba48cff4905bff72dec56f707646ccb1a4e928bc6afd8af"
Nov 02 02:13:37.430585 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.430507    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c664d046bbc02d4eba48cff4905bff72dec56f707646ccb1a4e928bc6afd8af"} err="failed to get container status \"3c664d046bbc02d4eba48cff4905bff72dec56f707646ccb1a4e928bc6afd8af\": rpc error: code = NotFound desc = could not find container \"3c664d046bbc02d4eba48cff4905bff72dec56f707646ccb1a4e928bc6afd8af\": container with ID starting with 3c664d046bbc02d4eba48cff4905bff72dec56f707646ccb1a4e928bc6afd8af not found: ID does not exist"
Nov 02 02:13:37.574628 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.574592    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8"]
Nov 02 02:13:37.580123 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.580095    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-257466-7579c4945f-7qdb8"]
Nov 02 02:13:37.688201 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.688117    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg"]
Nov 02 02:13:37.692452 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:37.692427    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-257466-7c449d7795-n6tkg"]
Nov 02 02:13:38.993594 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:38.993562    5639 scope.go:122] "RemoveContainer" containerID="0f81d2058f0c2535159fe2841192680618c8d48c8a55c1cda272327323733bdb"
Nov 02 02:13:39.002747 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:39.002726    5639 scope.go:122] "RemoveContainer" containerID="feb27bcb44c3e8b8ec2d3cc84f6eb88cbc40bcaf26ee8b001eb0ccb2f2cbea4e"
Nov 02 02:13:39.011572 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:39.011552    5639 scope.go:122] "RemoveContainer" containerID="dc2eb100db07c74aeaf9b624c9be3ef508d8b8dd26b870e4965dfb958b17f989"
Nov 02 02:13:39.020225 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:39.020200    5639 scope.go:122] "RemoveContainer" containerID="dd1f93b06b8b3634454b6d3b837acb966eac000cfe561a349476331be66885f2"
Nov 02 02:13:39.028707 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:39.028687    5639 scope.go:122] "RemoveContainer" containerID="023045ad613c6121fbbbd5069a6677a2a60dc61df52229c56833c0b58238607c"
Nov 02 02:13:39.217034 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:39.216999    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8e14587-d00a-43a2-8eba-b907a34a3d91" path="/var/lib/kubelet/pods/d8e14587-d00a-43a2-8eba-b907a34a3d91/volumes"
Nov 02 02:13:39.217592 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:39.217576    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f35a24df-cd1c-4768-9596-3ef625d19864" path="/var/lib/kubelet/pods/f35a24df-cd1c-4768-9596-3ef625d19864/volumes"
Nov 02 02:13:43.476054 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:43.476022    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"]
Nov 02 02:13:43.476461 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:43.476253    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g" podUID="70900cc0-155e-46f3-bf31-14169d79b332" containerName="istio-proxy" containerID="cri-o://d9e37ccbf6a89c8ecc79d81a2498e58f3bdfceb69c2e50f17ddd79db6fcb0e5c" gracePeriod=30
Nov 02 02:13:44.353970 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.353935    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd"]
Nov 02 02:13:44.354474 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.354458    5639 cpu_manager.go:414] "containerMap: removing container" podUID="d8e14587-d00a-43a2-8eba-b907a34a3d91" containerName="payload-processing"
Nov 02 02:13:44.354548 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.354475    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8e14587-d00a-43a2-8eba-b907a34a3d91" containerName="payload-processing"
Nov 02 02:13:44.354548 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.354490    5639 cpu_manager.go:414] "containerMap: removing container" podUID="b4f337c9-0884-424c-934d-0a1015586045" containerName="revoke-keys"
Nov 02 02:13:44.354548 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.354495    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4f337c9-0884-424c-934d-0a1015586045" containerName="revoke-keys"
Nov 02 02:13:44.354548 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.354510    5639 cpu_manager.go:414] "containerMap: removing container" podUID="f35a24df-cd1c-4768-9596-3ef625d19864" containerName="payload-pre-processing"
Nov 02 02:13:44.354548 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.354516    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f35a24df-cd1c-4768-9596-3ef625d19864" containerName="payload-pre-processing"
Nov 02 02:13:44.354548 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.354534    5639 cpu_manager.go:414] "containerMap: removing container" podUID="2e47ced9-b31c-4196-8b5c-0e980cf9297e" containerName="istio-proxy"
Nov 02 02:13:44.354548 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.354538    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e47ced9-b31c-4196-8b5c-0e980cf9297e" containerName="istio-proxy"
Nov 02 02:13:44.354548 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.354550    5639 cpu_manager.go:414] "containerMap: removing container" podUID="2e47ced9-b31c-4196-8b5c-0e980cf9297e" containerName="istio-proxy"
Nov 02 02:13:44.354844 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.354555    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e47ced9-b31c-4196-8b5c-0e980cf9297e" containerName="istio-proxy"
Nov 02 02:13:44.354844 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.354621    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8e14587-d00a-43a2-8eba-b907a34a3d91" containerName="payload-processing"
Nov 02 02:13:44.354844 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.354629    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4f337c9-0884-424c-934d-0a1015586045" containerName="revoke-keys"
Nov 02 02:13:44.354844 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.354637    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="f35a24df-cd1c-4768-9596-3ef625d19864" containerName="payload-pre-processing"
Nov 02 02:13:44.354844 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.354646    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e47ced9-b31c-4196-8b5c-0e980cf9297e" containerName="istio-proxy"
Nov 02 02:13:44.354844 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.354653    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e47ced9-b31c-4196-8b5c-0e980cf9297e" containerName="istio-proxy"
Nov 02 02:13:44.359922 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.359903    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd"
Nov 02 02:13:44.412007 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.411969    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:13:44.454901 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.454869    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf7tc\" (UniqueName: \"kubernetes.io/projected/ceefe2e0-68cc-427a-b8db-b534036155da-kube-api-access-kf7tc\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd\" (UID: \"ceefe2e0-68cc-427a-b8db-b534036155da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd"
Nov 02 02:13:44.455068 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.455035    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceefe2e0-68cc-427a-b8db-b534036155da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd\" (UID: \"ceefe2e0-68cc-427a-b8db-b534036155da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd"
Nov 02 02:13:44.455163 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.455137    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceefe2e0-68cc-427a-b8db-b534036155da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd\" (UID: \"ceefe2e0-68cc-427a-b8db-b534036155da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd"
Nov 02 02:13:44.461543 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.461511    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:13:44.461705 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.461650    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:13:44.461750 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.461740    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:13:44.463886 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.463868    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd"]
Nov 02 02:13:44.556070 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.556028    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceefe2e0-68cc-427a-b8db-b534036155da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd\" (UID: \"ceefe2e0-68cc-427a-b8db-b534036155da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd"
Nov 02 02:13:44.556555 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.556081    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceefe2e0-68cc-427a-b8db-b534036155da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd\" (UID: \"ceefe2e0-68cc-427a-b8db-b534036155da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd"
Nov 02 02:13:44.556555 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.556139    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf7tc\" (UniqueName: \"kubernetes.io/projected/ceefe2e0-68cc-427a-b8db-b534036155da-kube-api-access-kf7tc\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd\" (UID: \"ceefe2e0-68cc-427a-b8db-b534036155da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd"
Nov 02 02:13:44.556735 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.556713    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceefe2e0-68cc-427a-b8db-b534036155da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd\" (UID: \"ceefe2e0-68cc-427a-b8db-b534036155da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd"
Nov 02 02:13:44.558620 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.558602    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceefe2e0-68cc-427a-b8db-b534036155da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd\" (UID: \"ceefe2e0-68cc-427a-b8db-b534036155da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd"
Nov 02 02:13:44.617732 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.617663    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf7tc\" (UniqueName: \"kubernetes.io/projected/ceefe2e0-68cc-427a-b8db-b534036155da-kube-api-access-kf7tc\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd\" (UID: \"ceefe2e0-68cc-427a-b8db-b534036155da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd"
Nov 02 02:13:44.673169 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:44.673140    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd"
Nov 02 02:13:45.001973 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:45.001881    5639 patch_prober.go:29] interesting pod/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:13:45.002146 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:45.001965    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g" podUID="70900cc0-155e-46f3-bf31-14169d79b332" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:13:45.045948 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:45.045920    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd"]
Nov 02 02:13:45.047971 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:13:45.047941    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceefe2e0_68cc_427a_b8db_b534036155da.slice/crio-4c066f2aceb5bbfb4728db26b6bc752ec2b9862633677c489698b102b7287637 WatchSource:0}: Error finding container 4c066f2aceb5bbfb4728db26b6bc752ec2b9862633677c489698b102b7287637: Status 404 returned error can't find the container with id 4c066f2aceb5bbfb4728db26b6bc752ec2b9862633677c489698b102b7287637
Nov 02 02:13:45.466568 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:45.466530    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd" event={"ID":"ceefe2e0-68cc-427a-b8db-b534036155da","Type":"ContainerStarted","Data":"fb75dded064da9124e8285d0bdc3dac25437109d1e18ae17de7a2287dd875234"}
Nov 02 02:13:45.466761 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:45.466588    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd" event={"ID":"ceefe2e0-68cc-427a-b8db-b534036155da","Type":"ContainerStarted","Data":"4c066f2aceb5bbfb4728db26b6bc752ec2b9862633677c489698b102b7287637"}
Nov 02 02:13:45.583085 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:45.583033    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd" podStartSLOduration=1.58301867 podStartE2EDuration="1.58301867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:13:45.581475632 +0000 UTC m=+2890.485157745" watchObservedRunningTime="2026-11-02 02:13:45.58301867 +0000 UTC m=+2890.486700833"
Nov 02 02:13:46.473392 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:46.473359    5639 generic.go:356] "Generic (PLEG): container finished" podID="ceefe2e0-68cc-427a-b8db-b534036155da" containerID="fb75dded064da9124e8285d0bdc3dac25437109d1e18ae17de7a2287dd875234" exitCode=0
Nov 02 02:13:46.473581 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:46.473415    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd" event={"ID":"ceefe2e0-68cc-427a-b8db-b534036155da","Type":"ContainerDied","Data":"fb75dded064da9124e8285d0bdc3dac25437109d1e18ae17de7a2287dd875234"}
Nov 02 02:13:47.612139 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:47.612114    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd"
Nov 02 02:13:47.685412 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:47.685383    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceefe2e0-68cc-427a-b8db-b534036155da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceefe2e0-68cc-427a-b8db-b534036155da-maas-api-xks-ca\") pod \"ceefe2e0-68cc-427a-b8db-b534036155da\" (UID: \"ceefe2e0-68cc-427a-b8db-b534036155da\") "
Nov 02 02:13:47.685577 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:47.685429    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceefe2e0-68cc-427a-b8db-b534036155da-kube-api-access-kf7tc\" (UniqueName: \"kubernetes.io/projected/ceefe2e0-68cc-427a-b8db-b534036155da-kube-api-access-kf7tc\") pod \"ceefe2e0-68cc-427a-b8db-b534036155da\" (UID: \"ceefe2e0-68cc-427a-b8db-b534036155da\") "
Nov 02 02:13:47.685577 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:47.685538    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceefe2e0-68cc-427a-b8db-b534036155da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceefe2e0-68cc-427a-b8db-b534036155da-maas-api-openshift-ca\") pod \"ceefe2e0-68cc-427a-b8db-b534036155da\" (UID: \"ceefe2e0-68cc-427a-b8db-b534036155da\") "
Nov 02 02:13:47.685913 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:47.685893    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceefe2e0-68cc-427a-b8db-b534036155da-maas-api-openshift-ca" pod "ceefe2e0-68cc-427a-b8db-b534036155da" (UID: "ceefe2e0-68cc-427a-b8db-b534036155da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:13:47.687745 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:47.687721    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceefe2e0-68cc-427a-b8db-b534036155da-maas-api-xks-ca" pod "ceefe2e0-68cc-427a-b8db-b534036155da" (UID: "ceefe2e0-68cc-427a-b8db-b534036155da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:13:47.687799 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:47.687778    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceefe2e0-68cc-427a-b8db-b534036155da-kube-api-access-kf7tc" pod "ceefe2e0-68cc-427a-b8db-b534036155da" (UID: "ceefe2e0-68cc-427a-b8db-b534036155da"). InnerVolumeSpecName "kube-api-access-kf7tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:13:47.786907 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:47.786828    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceefe2e0-68cc-427a-b8db-b534036155da-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:47.786907 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:47.786856    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceefe2e0-68cc-427a-b8db-b534036155da-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:47.786907 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:47.786869    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf7tc\" (UniqueName: \"kubernetes.io/projected/ceefe2e0-68cc-427a-b8db-b534036155da-kube-api-access-kf7tc\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:48.486258 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.486220    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd"
Nov 02 02:13:48.486488 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.486255    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd" event={"ID":"ceefe2e0-68cc-427a-b8db-b534036155da","Type":"ContainerDied","Data":"4c066f2aceb5bbfb4728db26b6bc752ec2b9862633677c489698b102b7287637"}
Nov 02 02:13:48.486488 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.486330    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c066f2aceb5bbfb4728db26b6bc752ec2b9862633677c489698b102b7287637"
Nov 02 02:13:48.735800 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.735772    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 02:13:48.796464 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.796372    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70900cc0-155e-46f3-bf31-14169d79b332-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70900cc0-155e-46f3-bf31-14169d79b332-istiod-ca-cert\") pod \"70900cc0-155e-46f3-bf31-14169d79b332\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") "
Nov 02 02:13:48.796464 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.796444    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-credential-socket\") pod \"70900cc0-155e-46f3-bf31-14169d79b332\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") "
Nov 02 02:13:48.796702 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.796488    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-workload-certs\") pod \"70900cc0-155e-46f3-bf31-14169d79b332\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") "
Nov 02 02:13:48.796702 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.796520    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70900cc0-155e-46f3-bf31-14169d79b332-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70900cc0-155e-46f3-bf31-14169d79b332-istio-podinfo\") pod \"70900cc0-155e-46f3-bf31-14169d79b332\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") "
Nov 02 02:13:48.796702 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.796584    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-istio-data\") pod \"70900cc0-155e-46f3-bf31-14169d79b332\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") "
Nov 02 02:13:48.796702 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.796661    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70900cc0-155e-46f3-bf31-14169d79b332-istio-token\" (UniqueName: \"kubernetes.io/projected/70900cc0-155e-46f3-bf31-14169d79b332-istio-token\") pod \"70900cc0-155e-46f3-bf31-14169d79b332\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") "
Nov 02 02:13:48.796702 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.796687    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-istio-envoy\") pod \"70900cc0-155e-46f3-bf31-14169d79b332\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") "
Nov 02 02:13:48.796981 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.796763    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70900cc0-155e-46f3-bf31-14169d79b332-kube-api-access-pjb7c\" (UniqueName: \"kubernetes.io/projected/70900cc0-155e-46f3-bf31-14169d79b332-kube-api-access-pjb7c\") pod \"70900cc0-155e-46f3-bf31-14169d79b332\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") "
Nov 02 02:13:48.796981 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.796778    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-workload-certs" pod "70900cc0-155e-46f3-bf31-14169d79b332" (UID: "70900cc0-155e-46f3-bf31-14169d79b332"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:13:48.796981 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.796795    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-credential-socket" pod "70900cc0-155e-46f3-bf31-14169d79b332" (UID: "70900cc0-155e-46f3-bf31-14169d79b332"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:13:48.796981 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.796791    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-workload-socket\") pod \"70900cc0-155e-46f3-bf31-14169d79b332\" (UID: \"70900cc0-155e-46f3-bf31-14169d79b332\") "
Nov 02 02:13:48.796981 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.796836    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70900cc0-155e-46f3-bf31-14169d79b332-istiod-ca-cert" pod "70900cc0-155e-46f3-bf31-14169d79b332" (UID: "70900cc0-155e-46f3-bf31-14169d79b332"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:13:48.797246 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.797015    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-workload-socket" pod "70900cc0-155e-46f3-bf31-14169d79b332" (UID: "70900cc0-155e-46f3-bf31-14169d79b332"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:13:48.797246 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.797230    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-istio-data" pod "70900cc0-155e-46f3-bf31-14169d79b332" (UID: "70900cc0-155e-46f3-bf31-14169d79b332"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:13:48.797582 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.797504    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:48.797582 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.797530    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70900cc0-155e-46f3-bf31-14169d79b332-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:48.797582 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.797544    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:48.797582 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.797556    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:48.797582 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.797568    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:48.798966 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.798933    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70900cc0-155e-46f3-bf31-14169d79b332-istio-podinfo" pod "70900cc0-155e-46f3-bf31-14169d79b332" (UID: "70900cc0-155e-46f3-bf31-14169d79b332"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:13:48.799081 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.799012    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70900cc0-155e-46f3-bf31-14169d79b332-istio-token" pod "70900cc0-155e-46f3-bf31-14169d79b332" (UID: "70900cc0-155e-46f3-bf31-14169d79b332"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:13:48.799081 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.799022    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-istio-envoy" pod "70900cc0-155e-46f3-bf31-14169d79b332" (UID: "70900cc0-155e-46f3-bf31-14169d79b332"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:13:48.799153 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.799081    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70900cc0-155e-46f3-bf31-14169d79b332-kube-api-access-pjb7c" pod "70900cc0-155e-46f3-bf31-14169d79b332" (UID: "70900cc0-155e-46f3-bf31-14169d79b332"). InnerVolumeSpecName "kube-api-access-pjb7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:13:48.898711 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.898680    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70900cc0-155e-46f3-bf31-14169d79b332-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:48.898711 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.898706    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70900cc0-155e-46f3-bf31-14169d79b332-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:48.898711 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.898718    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjb7c\" (UniqueName: \"kubernetes.io/projected/70900cc0-155e-46f3-bf31-14169d79b332-kube-api-access-pjb7c\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:48.898943 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:48.898728    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70900cc0-155e-46f3-bf31-14169d79b332-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:49.494017 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:49.493914    5639 generic.go:356] "Generic (PLEG): container finished" podID="70900cc0-155e-46f3-bf31-14169d79b332" containerID="d9e37ccbf6a89c8ecc79d81a2498e58f3bdfceb69c2e50f17ddd79db6fcb0e5c" exitCode=0
Nov 02 02:13:49.494017 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:49.493981    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"
Nov 02 02:13:49.494226 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:49.494019    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g" event={"ID":"70900cc0-155e-46f3-bf31-14169d79b332","Type":"ContainerDied","Data":"d9e37ccbf6a89c8ecc79d81a2498e58f3bdfceb69c2e50f17ddd79db6fcb0e5c"}
Nov 02 02:13:49.494226 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:49.494073    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g" event={"ID":"70900cc0-155e-46f3-bf31-14169d79b332","Type":"ContainerDied","Data":"cc69585e677bc597ba1fc5ab1539461722a00699266428e75a319d111a0b43bb"}
Nov 02 02:13:49.494226 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:49.494098    5639 scope.go:122] "RemoveContainer" containerID="d9e37ccbf6a89c8ecc79d81a2498e58f3bdfceb69c2e50f17ddd79db6fcb0e5c"
Nov 02 02:13:49.504156 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:49.504128    5639 scope.go:122] "RemoveContainer" containerID="8964a3324e240077aca0c73deb70a63366b0077d5e482387449a2cedc0b79ca5"
Nov 02 02:13:49.513346 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:49.513324    5639 scope.go:122] "RemoveContainer" containerID="d9e37ccbf6a89c8ecc79d81a2498e58f3bdfceb69c2e50f17ddd79db6fcb0e5c"
Nov 02 02:13:49.513639 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:13:49.513620    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9e37ccbf6a89c8ecc79d81a2498e58f3bdfceb69c2e50f17ddd79db6fcb0e5c\": container with ID starting with d9e37ccbf6a89c8ecc79d81a2498e58f3bdfceb69c2e50f17ddd79db6fcb0e5c not found: ID does not exist" containerID="d9e37ccbf6a89c8ecc79d81a2498e58f3bdfceb69c2e50f17ddd79db6fcb0e5c"
Nov 02 02:13:49.513685 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:49.513648    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9e37ccbf6a89c8ecc79d81a2498e58f3bdfceb69c2e50f17ddd79db6fcb0e5c"} err="failed to get container status \"d9e37ccbf6a89c8ecc79d81a2498e58f3bdfceb69c2e50f17ddd79db6fcb0e5c\": rpc error: code = NotFound desc = could not find container \"d9e37ccbf6a89c8ecc79d81a2498e58f3bdfceb69c2e50f17ddd79db6fcb0e5c\": container with ID starting with d9e37ccbf6a89c8ecc79d81a2498e58f3bdfceb69c2e50f17ddd79db6fcb0e5c not found: ID does not exist"
Nov 02 02:13:49.513685 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:49.513665    5639 scope.go:122] "RemoveContainer" containerID="8964a3324e240077aca0c73deb70a63366b0077d5e482387449a2cedc0b79ca5"
Nov 02 02:13:49.513934 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:13:49.513917    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8964a3324e240077aca0c73deb70a63366b0077d5e482387449a2cedc0b79ca5\": container with ID starting with 8964a3324e240077aca0c73deb70a63366b0077d5e482387449a2cedc0b79ca5 not found: ID does not exist" containerID="8964a3324e240077aca0c73deb70a63366b0077d5e482387449a2cedc0b79ca5"
Nov 02 02:13:49.513991 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:49.513940    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8964a3324e240077aca0c73deb70a63366b0077d5e482387449a2cedc0b79ca5"} err="failed to get container status \"8964a3324e240077aca0c73deb70a63366b0077d5e482387449a2cedc0b79ca5\": rpc error: code = NotFound desc = could not find container \"8964a3324e240077aca0c73deb70a63366b0077d5e482387449a2cedc0b79ca5\": container with ID starting with 8964a3324e240077aca0c73deb70a63366b0077d5e482387449a2cedc0b79ca5 not found: ID does not exist"
Nov 02 02:13:49.659877 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:49.659842    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"]
Nov 02 02:13:49.709987 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:49.709953    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-257466-openshift-default-79678c79f7-4n65g"]
Nov 02 02:13:51.217269 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:51.217233    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70900cc0-155e-46f3-bf31-14169d79b332" path="/var/lib/kubelet/pods/70900cc0-155e-46f3-bf31-14169d79b332/volumes"
Nov 02 02:13:55.828323 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:55.828288    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc"]
Nov 02 02:13:55.937663 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:13:55.937624    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x"]
Nov 02 02:13:57.419815 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:13:57.419774    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:14:00.829627 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:00.829588    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc" podUID="b1edf1cd-ee67-4762-9093-3e41dbd9df4c" containerName="payload-processing" containerID="cri-o://67504323135cb0acb763b72d7f06d48baa127f9560e87724108e85e809cdc68d" gracePeriod=25
Nov 02 02:14:00.938529 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:00.938492    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x" podUID="6f21df8d-e07e-46d4-9186-408789552242" containerName="payload-pre-processing" containerID="cri-o://6377b8a4ef4cba6ea7c5525d67f9a926258599dd595eb0bb33d3317f2b51976e" gracePeriod=25
Nov 02 02:14:01.071231 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.071206    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc"
Nov 02 02:14:01.196960 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.196931    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x"
Nov 02 02:14:01.220670 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.220645    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1edf1cd-ee67-4762-9093-3e41dbd9df4c-kube-api-access-znxcx\" (UniqueName: \"kubernetes.io/projected/b1edf1cd-ee67-4762-9093-3e41dbd9df4c-kube-api-access-znxcx\") pod \"b1edf1cd-ee67-4762-9093-3e41dbd9df4c\" (UID: \"b1edf1cd-ee67-4762-9093-3e41dbd9df4c\") "
Nov 02 02:14:01.220840 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.220737    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1edf1cd-ee67-4762-9093-3e41dbd9df4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1edf1cd-ee67-4762-9093-3e41dbd9df4c-plugins-config-volume\") pod \"b1edf1cd-ee67-4762-9093-3e41dbd9df4c\" (UID: \"b1edf1cd-ee67-4762-9093-3e41dbd9df4c\") "
Nov 02 02:14:01.221068 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.221046    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1edf1cd-ee67-4762-9093-3e41dbd9df4c-plugins-config-volume" pod "b1edf1cd-ee67-4762-9093-3e41dbd9df4c" (UID: "b1edf1cd-ee67-4762-9093-3e41dbd9df4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:14:01.223048 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.223022    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1edf1cd-ee67-4762-9093-3e41dbd9df4c-kube-api-access-znxcx" pod "b1edf1cd-ee67-4762-9093-3e41dbd9df4c" (UID: "b1edf1cd-ee67-4762-9093-3e41dbd9df4c"). InnerVolumeSpecName "kube-api-access-znxcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:14:01.321343 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.321306    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f21df8d-e07e-46d4-9186-408789552242-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f21df8d-e07e-46d4-9186-408789552242-plugins-config-volume\") pod \"6f21df8d-e07e-46d4-9186-408789552242\" (UID: \"6f21df8d-e07e-46d4-9186-408789552242\") "
Nov 02 02:14:01.321531 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.321475    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f21df8d-e07e-46d4-9186-408789552242-kube-api-access-zscsv\" (UniqueName: \"kubernetes.io/projected/6f21df8d-e07e-46d4-9186-408789552242-kube-api-access-zscsv\") pod \"6f21df8d-e07e-46d4-9186-408789552242\" (UID: \"6f21df8d-e07e-46d4-9186-408789552242\") "
Nov 02 02:14:01.321721 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.321696    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f21df8d-e07e-46d4-9186-408789552242-plugins-config-volume" pod "6f21df8d-e07e-46d4-9186-408789552242" (UID: "6f21df8d-e07e-46d4-9186-408789552242"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:14:01.321846 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.321829    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1edf1cd-ee67-4762-9093-3e41dbd9df4c-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:01.321902 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.321853    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f21df8d-e07e-46d4-9186-408789552242-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:01.321902 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.321868    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znxcx\" (UniqueName: \"kubernetes.io/projected/b1edf1cd-ee67-4762-9093-3e41dbd9df4c-kube-api-access-znxcx\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:01.323757 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.323733    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f21df8d-e07e-46d4-9186-408789552242-kube-api-access-zscsv" pod "6f21df8d-e07e-46d4-9186-408789552242" (UID: "6f21df8d-e07e-46d4-9186-408789552242"). InnerVolumeSpecName "kube-api-access-zscsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:14:01.422505 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.422409    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zscsv\" (UniqueName: \"kubernetes.io/projected/6f21df8d-e07e-46d4-9186-408789552242-kube-api-access-zscsv\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:01.562406 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.562348    5639 generic.go:356] "Generic (PLEG): container finished" podID="b1edf1cd-ee67-4762-9093-3e41dbd9df4c" containerID="67504323135cb0acb763b72d7f06d48baa127f9560e87724108e85e809cdc68d" exitCode=0
Nov 02 02:14:01.562598 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.562416    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc"
Nov 02 02:14:01.562598 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.562430    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc" event={"ID":"b1edf1cd-ee67-4762-9093-3e41dbd9df4c","Type":"ContainerDied","Data":"67504323135cb0acb763b72d7f06d48baa127f9560e87724108e85e809cdc68d"}
Nov 02 02:14:01.562598 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.562484    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc" event={"ID":"b1edf1cd-ee67-4762-9093-3e41dbd9df4c","Type":"ContainerDied","Data":"0ea1d138954af04ed8f9d4a787f38b613fca45884873589f9a3e8240726e3c68"}
Nov 02 02:14:01.562598 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.562506    5639 scope.go:122] "RemoveContainer" containerID="67504323135cb0acb763b72d7f06d48baa127f9560e87724108e85e809cdc68d"
Nov 02 02:14:01.564244 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.564214    5639 generic.go:356] "Generic (PLEG): container finished" podID="6f21df8d-e07e-46d4-9186-408789552242" containerID="6377b8a4ef4cba6ea7c5525d67f9a926258599dd595eb0bb33d3317f2b51976e" exitCode=0
Nov 02 02:14:01.564400 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.564244    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x" event={"ID":"6f21df8d-e07e-46d4-9186-408789552242","Type":"ContainerDied","Data":"6377b8a4ef4cba6ea7c5525d67f9a926258599dd595eb0bb33d3317f2b51976e"}
Nov 02 02:14:01.564400 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.564272    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x"
Nov 02 02:14:01.564400 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.564324    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x" event={"ID":"6f21df8d-e07e-46d4-9186-408789552242","Type":"ContainerDied","Data":"ff441400d704c278a81999a094155dbf2abde9d0875b31592c2af6cb4c8c01b0"}
Nov 02 02:14:01.573345 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.573327    5639 scope.go:122] "RemoveContainer" containerID="67504323135cb0acb763b72d7f06d48baa127f9560e87724108e85e809cdc68d"
Nov 02 02:14:01.573652 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:14:01.573601    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67504323135cb0acb763b72d7f06d48baa127f9560e87724108e85e809cdc68d\": container with ID starting with 67504323135cb0acb763b72d7f06d48baa127f9560e87724108e85e809cdc68d not found: ID does not exist" containerID="67504323135cb0acb763b72d7f06d48baa127f9560e87724108e85e809cdc68d"
Nov 02 02:14:01.573652 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.573633    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67504323135cb0acb763b72d7f06d48baa127f9560e87724108e85e809cdc68d"} err="failed to get container status \"67504323135cb0acb763b72d7f06d48baa127f9560e87724108e85e809cdc68d\": rpc error: code = NotFound desc = could not find container \"67504323135cb0acb763b72d7f06d48baa127f9560e87724108e85e809cdc68d\": container with ID starting with 67504323135cb0acb763b72d7f06d48baa127f9560e87724108e85e809cdc68d not found: ID does not exist"
Nov 02 02:14:01.573781 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.573657    5639 scope.go:122] "RemoveContainer" containerID="6377b8a4ef4cba6ea7c5525d67f9a926258599dd595eb0bb33d3317f2b51976e"
Nov 02 02:14:01.582075 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.582057    5639 scope.go:122] "RemoveContainer" containerID="6377b8a4ef4cba6ea7c5525d67f9a926258599dd595eb0bb33d3317f2b51976e"
Nov 02 02:14:01.582469 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:14:01.582448    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6377b8a4ef4cba6ea7c5525d67f9a926258599dd595eb0bb33d3317f2b51976e\": container with ID starting with 6377b8a4ef4cba6ea7c5525d67f9a926258599dd595eb0bb33d3317f2b51976e not found: ID does not exist" containerID="6377b8a4ef4cba6ea7c5525d67f9a926258599dd595eb0bb33d3317f2b51976e"
Nov 02 02:14:01.582539 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.582477    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6377b8a4ef4cba6ea7c5525d67f9a926258599dd595eb0bb33d3317f2b51976e"} err="failed to get container status \"6377b8a4ef4cba6ea7c5525d67f9a926258599dd595eb0bb33d3317f2b51976e\": rpc error: code = NotFound desc = could not find container \"6377b8a4ef4cba6ea7c5525d67f9a926258599dd595eb0bb33d3317f2b51976e\": container with ID starting with 6377b8a4ef4cba6ea7c5525d67f9a926258599dd595eb0bb33d3317f2b51976e not found: ID does not exist"
Nov 02 02:14:01.727227 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.727146    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x"]
Nov 02 02:14:01.733378 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.733355    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-39ceb2-785f94879-nd25x"]
Nov 02 02:14:01.843344 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.843313    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc"]
Nov 02 02:14:01.848783 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:01.848758    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-39ceb2-74bdf8d6b9-q7rrc"]
Nov 02 02:14:03.218441 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:03.218412    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f21df8d-e07e-46d4-9186-408789552242" path="/var/lib/kubelet/pods/6f21df8d-e07e-46d4-9186-408789552242/volumes"
Nov 02 02:14:03.218916 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:03.218791    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1edf1cd-ee67-4762-9093-3e41dbd9df4c" path="/var/lib/kubelet/pods/b1edf1cd-ee67-4762-9093-3e41dbd9df4c/volumes"
Nov 02 02:14:09.204046 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:09.204014    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd"]
Nov 02 02:14:09.254358 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:09.254323    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-39ceb2-8vfdd"]
Nov 02 02:14:10.707009 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:10.706974    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:14:11.216901 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:11.216872    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceefe2e0-68cc-427a-b8db-b534036155da" path="/var/lib/kubelet/pods/ceefe2e0-68cc-427a-b8db-b534036155da/volumes"
Nov 02 02:14:16.001320 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:16.001267    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"]
Nov 02 02:14:16.001758 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:16.001499    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z" podUID="5d218ddb-88c3-4646-a31b-cdd5d05dbc1d" containerName="istio-proxy" containerID="cri-o://9b967f9b1fb4766f6dfc5d32191bc725ee918927b0a17b8a49dead9b257cf89a" gracePeriod=30
Nov 02 02:14:16.979162 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:16.979118    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr"]
Nov 02 02:14:16.979836 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:16.979807    5639 cpu_manager.go:414] "containerMap: removing container" podUID="b1edf1cd-ee67-4762-9093-3e41dbd9df4c" containerName="payload-processing"
Nov 02 02:14:16.979836 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:16.979834    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1edf1cd-ee67-4762-9093-3e41dbd9df4c" containerName="payload-processing"
Nov 02 02:14:16.980012 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:16.979867    5639 cpu_manager.go:414] "containerMap: removing container" podUID="6f21df8d-e07e-46d4-9186-408789552242" containerName="payload-pre-processing"
Nov 02 02:14:16.980012 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:16.979875    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f21df8d-e07e-46d4-9186-408789552242" containerName="payload-pre-processing"
Nov 02 02:14:16.980012 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:16.979891    5639 cpu_manager.go:414] "containerMap: removing container" podUID="ceefe2e0-68cc-427a-b8db-b534036155da" containerName="revoke-keys"
Nov 02 02:14:16.980012 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:16.979897    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceefe2e0-68cc-427a-b8db-b534036155da" containerName="revoke-keys"
Nov 02 02:14:16.980012 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:16.979931    5639 cpu_manager.go:414] "containerMap: removing container" podUID="70900cc0-155e-46f3-bf31-14169d79b332" containerName="istio-proxy"
Nov 02 02:14:16.980012 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:16.979937    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70900cc0-155e-46f3-bf31-14169d79b332" containerName="istio-proxy"
Nov 02 02:14:16.980012 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:16.979952    5639 cpu_manager.go:414] "containerMap: removing container" podUID="70900cc0-155e-46f3-bf31-14169d79b332" containerName="istio-proxy"
Nov 02 02:14:16.980012 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:16.979958    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70900cc0-155e-46f3-bf31-14169d79b332" containerName="istio-proxy"
Nov 02 02:14:16.980265 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:16.980055    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f21df8d-e07e-46d4-9186-408789552242" containerName="payload-pre-processing"
Nov 02 02:14:16.980265 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:16.980065    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceefe2e0-68cc-427a-b8db-b534036155da" containerName="revoke-keys"
Nov 02 02:14:16.980265 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:16.980082    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="70900cc0-155e-46f3-bf31-14169d79b332" containerName="istio-proxy"
Nov 02 02:14:16.980265 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:16.980093    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="70900cc0-155e-46f3-bf31-14169d79b332" containerName="istio-proxy"
Nov 02 02:14:16.980265 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:16.980103    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1edf1cd-ee67-4762-9093-3e41dbd9df4c" containerName="payload-processing"
Nov 02 02:14:16.983796 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:16.983773    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr"
Nov 02 02:14:17.040782 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:17.040743    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr"]
Nov 02 02:14:17.080460 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:17.080427    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m95wp\" (UniqueName: \"kubernetes.io/projected/6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3-kube-api-access-m95wp\") pod \"maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr\" (UID: \"6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr"
Nov 02 02:14:17.080460 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:17.080463    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr\" (UID: \"6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr"
Nov 02 02:14:17.080708 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:17.080486    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr\" (UID: \"6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr"
Nov 02 02:14:17.088502 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:17.088454    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:14:17.088502 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:17.088500    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:14:17.088721 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:17.088511    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:14:17.088721 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:17.088543    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:14:17.181532 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:17.181494    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m95wp\" (UniqueName: \"kubernetes.io/projected/6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3-kube-api-access-m95wp\") pod \"maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr\" (UID: \"6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr"
Nov 02 02:14:17.181532 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:17.181535    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr\" (UID: \"6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr"
Nov 02 02:14:17.181808 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:17.181553    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr\" (UID: \"6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr"
Nov 02 02:14:17.182196 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:17.182171    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr\" (UID: \"6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr"
Nov 02 02:14:17.184243 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:17.184220    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr\" (UID: \"6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr"
Nov 02 02:14:17.242877 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:17.242811    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m95wp\" (UniqueName: \"kubernetes.io/projected/6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3-kube-api-access-m95wp\") pod \"maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr\" (UID: \"6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr"
Nov 02 02:14:17.297913 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:17.297875    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr"
Nov 02 02:14:17.683629 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:17.683596    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr"]
Nov 02 02:14:17.685883 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:14:17.685857    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6359e29f_3c9d_4192_8c6e_7b1a13b4a2a3.slice/crio-b1926dc4be9c7d1e525fd349fc4884ae402049151958304d83e5eb344517ccab WatchSource:0}: Error finding container b1926dc4be9c7d1e525fd349fc4884ae402049151958304d83e5eb344517ccab: Status 404 returned error can't find the container with id b1926dc4be9c7d1e525fd349fc4884ae402049151958304d83e5eb344517ccab
Nov 02 02:14:18.658130 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:18.658096    5639 generic.go:356] "Generic (PLEG): container finished" podID="6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3" containerID="dd21238e36806988c9523009c03b951566a4c4b45ca916307c7a6865e8558db7" exitCode=0
Nov 02 02:14:18.658574 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:18.658188    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr" event={"ID":"6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3","Type":"ContainerDied","Data":"dd21238e36806988c9523009c03b951566a4c4b45ca916307c7a6865e8558db7"}
Nov 02 02:14:18.658574 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:18.658240    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr" event={"ID":"6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3","Type":"ContainerStarted","Data":"b1926dc4be9c7d1e525fd349fc4884ae402049151958304d83e5eb344517ccab"}
Nov 02 02:14:19.609112 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:19.609066    5639 patch_prober.go:29] interesting pod/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:14:19.609319 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:19.609136    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z" podUID="5d218ddb-88c3-4646-a31b-cdd5d05dbc1d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:14:19.798298 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:19.798262    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr"
Nov 02 02:14:19.909637 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:19.909549    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3-maas-api-openshift-ca\") pod \"6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3\" (UID: \"6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3\") "
Nov 02 02:14:19.909637 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:19.909614    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3-maas-api-xks-ca\") pod \"6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3\" (UID: \"6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3\") "
Nov 02 02:14:19.909838 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:19.909678    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3-kube-api-access-m95wp\" (UniqueName: \"kubernetes.io/projected/6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3-kube-api-access-m95wp\") pod \"6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3\" (UID: \"6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3\") "
Nov 02 02:14:19.910017 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:19.909989    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3-maas-api-openshift-ca" pod "6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3" (UID: "6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:14:19.911913 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:19.911882    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3-maas-api-xks-ca" pod "6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3" (UID: "6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:14:19.912047 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:19.911995    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3-kube-api-access-m95wp" pod "6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3" (UID: "6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3"). InnerVolumeSpecName "kube-api-access-m95wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:14:20.010729 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:20.010693    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m95wp\" (UniqueName: \"kubernetes.io/projected/6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3-kube-api-access-m95wp\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:20.010729 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:20.010723    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:20.010729 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:20.010733    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:20.672059 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:20.672028    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr"
Nov 02 02:14:20.672059 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:20.672045    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr" event={"ID":"6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3","Type":"ContainerDied","Data":"b1926dc4be9c7d1e525fd349fc4884ae402049151958304d83e5eb344517ccab"}
Nov 02 02:14:20.672327 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:20.672099    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1926dc4be9c7d1e525fd349fc4884ae402049151958304d83e5eb344517ccab"
Nov 02 02:14:21.256967 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.256943    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 02:14:21.322497 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.322417    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istiod-ca-cert\") pod \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") "
Nov 02 02:14:21.322497 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.322452    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-credential-socket\") pod \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") "
Nov 02 02:14:21.322497 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.322490    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-workload-socket\") pod \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") "
Nov 02 02:14:21.322784 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.322570    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-data\") pod \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") "
Nov 02 02:14:21.322784 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.322636    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-token\" (UniqueName: \"kubernetes.io/projected/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-token\") pod \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") "
Nov 02 02:14:21.322784 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.322672    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-envoy\") pod \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") "
Nov 02 02:14:21.322784 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.322703    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-kube-api-access-8nvxp\" (UniqueName: \"kubernetes.io/projected/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-kube-api-access-8nvxp\") pod \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") "
Nov 02 02:14:21.322784 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.322745    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-workload-certs\") pod \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") "
Nov 02 02:14:21.322784 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.322749    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-credential-socket" pod "5d218ddb-88c3-4646-a31b-cdd5d05dbc1d" (UID: "5d218ddb-88c3-4646-a31b-cdd5d05dbc1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:21.323107 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.322794    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-podinfo\") pod \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\" (UID: \"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d\") "
Nov 02 02:14:21.323107 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.322829    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-workload-socket" pod "5d218ddb-88c3-4646-a31b-cdd5d05dbc1d" (UID: "5d218ddb-88c3-4646-a31b-cdd5d05dbc1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:21.323107 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.322919    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-data" pod "5d218ddb-88c3-4646-a31b-cdd5d05dbc1d" (UID: "5d218ddb-88c3-4646-a31b-cdd5d05dbc1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:21.323107 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.323002    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-workload-certs" pod "5d218ddb-88c3-4646-a31b-cdd5d05dbc1d" (UID: "5d218ddb-88c3-4646-a31b-cdd5d05dbc1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:21.323358 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.323116    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istiod-ca-cert" pod "5d218ddb-88c3-4646-a31b-cdd5d05dbc1d" (UID: "5d218ddb-88c3-4646-a31b-cdd5d05dbc1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:14:21.323536 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.323472    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:21.323536 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.323500    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:21.323536 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.323515    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:21.323536 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.323529    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:21.323536 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.323545    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:21.325273 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.325241    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-token" pod "5d218ddb-88c3-4646-a31b-cdd5d05dbc1d" (UID: "5d218ddb-88c3-4646-a31b-cdd5d05dbc1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:14:21.325414 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.325370    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-envoy" pod "5d218ddb-88c3-4646-a31b-cdd5d05dbc1d" (UID: "5d218ddb-88c3-4646-a31b-cdd5d05dbc1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:21.325735 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.325717    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-podinfo" pod "5d218ddb-88c3-4646-a31b-cdd5d05dbc1d" (UID: "5d218ddb-88c3-4646-a31b-cdd5d05dbc1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:14:21.325838 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.325821    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-kube-api-access-8nvxp" pod "5d218ddb-88c3-4646-a31b-cdd5d05dbc1d" (UID: "5d218ddb-88c3-4646-a31b-cdd5d05dbc1d"). InnerVolumeSpecName "kube-api-access-8nvxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:14:21.424817 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.424787    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:21.424817 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.424814    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:21.424817 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.424823    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:21.425068 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.424832    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nvxp\" (UniqueName: \"kubernetes.io/projected/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d-kube-api-access-8nvxp\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:21.428384 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.428357    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"]
Nov 02 02:14:21.679606 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.679569    5639 generic.go:356] "Generic (PLEG): container finished" podID="5d218ddb-88c3-4646-a31b-cdd5d05dbc1d" containerID="9b967f9b1fb4766f6dfc5d32191bc725ee918927b0a17b8a49dead9b257cf89a" exitCode=0
Nov 02 02:14:21.679817 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.679624    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"
Nov 02 02:14:21.679817 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.679658    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z" event={"ID":"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d","Type":"ContainerDied","Data":"9b967f9b1fb4766f6dfc5d32191bc725ee918927b0a17b8a49dead9b257cf89a"}
Nov 02 02:14:21.679817 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.679728    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z" event={"ID":"5d218ddb-88c3-4646-a31b-cdd5d05dbc1d","Type":"ContainerDied","Data":"72d722099259df493de585a32403456907fde98f72748e24dd1166dd6263ccd6"}
Nov 02 02:14:21.679817 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.679760    5639 scope.go:122] "RemoveContainer" containerID="9b967f9b1fb4766f6dfc5d32191bc725ee918927b0a17b8a49dead9b257cf89a"
Nov 02 02:14:21.689945 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.689927    5639 scope.go:122] "RemoveContainer" containerID="c7546833bfb3c3832e8221acfb033a02a8ff85aca3e16399304f8f04bad8afc0"
Nov 02 02:14:21.698776 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.698758    5639 scope.go:122] "RemoveContainer" containerID="9b967f9b1fb4766f6dfc5d32191bc725ee918927b0a17b8a49dead9b257cf89a"
Nov 02 02:14:21.699040 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:14:21.699016    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b967f9b1fb4766f6dfc5d32191bc725ee918927b0a17b8a49dead9b257cf89a\": container with ID starting with 9b967f9b1fb4766f6dfc5d32191bc725ee918927b0a17b8a49dead9b257cf89a not found: ID does not exist" containerID="9b967f9b1fb4766f6dfc5d32191bc725ee918927b0a17b8a49dead9b257cf89a"
Nov 02 02:14:21.699085 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.699051    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b967f9b1fb4766f6dfc5d32191bc725ee918927b0a17b8a49dead9b257cf89a"} err="failed to get container status \"9b967f9b1fb4766f6dfc5d32191bc725ee918927b0a17b8a49dead9b257cf89a\": rpc error: code = NotFound desc = could not find container \"9b967f9b1fb4766f6dfc5d32191bc725ee918927b0a17b8a49dead9b257cf89a\": container with ID starting with 9b967f9b1fb4766f6dfc5d32191bc725ee918927b0a17b8a49dead9b257cf89a not found: ID does not exist"
Nov 02 02:14:21.699085 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.699069    5639 scope.go:122] "RemoveContainer" containerID="c7546833bfb3c3832e8221acfb033a02a8ff85aca3e16399304f8f04bad8afc0"
Nov 02 02:14:21.699345 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:14:21.699322    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7546833bfb3c3832e8221acfb033a02a8ff85aca3e16399304f8f04bad8afc0\": container with ID starting with c7546833bfb3c3832e8221acfb033a02a8ff85aca3e16399304f8f04bad8afc0 not found: ID does not exist" containerID="c7546833bfb3c3832e8221acfb033a02a8ff85aca3e16399304f8f04bad8afc0"
Nov 02 02:14:21.699417 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.699358    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7546833bfb3c3832e8221acfb033a02a8ff85aca3e16399304f8f04bad8afc0"} err="failed to get container status \"c7546833bfb3c3832e8221acfb033a02a8ff85aca3e16399304f8f04bad8afc0\": rpc error: code = NotFound desc = could not find container \"c7546833bfb3c3832e8221acfb033a02a8ff85aca3e16399304f8f04bad8afc0\": container with ID starting with c7546833bfb3c3832e8221acfb033a02a8ff85aca3e16399304f8f04bad8afc0 not found: ID does not exist"
Nov 02 02:14:21.853530 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.853493    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"]
Nov 02 02:14:21.861530 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:21.861495    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-39ceb2-openshift-default-56d954f999-dp56z"]
Nov 02 02:14:23.217661 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:23.217627    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d218ddb-88c3-4646-a31b-cdd5d05dbc1d" path="/var/lib/kubelet/pods/5d218ddb-88c3-4646-a31b-cdd5d05dbc1d/volumes"
Nov 02 02:14:27.419625 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:14:27.419589    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:14:28.455157 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.455112    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5064232b-rwcz5"]
Nov 02 02:14:28.455785 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.455763    5639 cpu_manager.go:414] "containerMap: removing container" podUID="6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3" containerName="revoke-keys"
Nov 02 02:14:28.455883 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.455787    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3" containerName="revoke-keys"
Nov 02 02:14:28.455883 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.455805    5639 cpu_manager.go:414] "containerMap: removing container" podUID="5d218ddb-88c3-4646-a31b-cdd5d05dbc1d" containerName="istio-proxy"
Nov 02 02:14:28.455883 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.455812    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d218ddb-88c3-4646-a31b-cdd5d05dbc1d" containerName="istio-proxy"
Nov 02 02:14:28.455883 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.455853    5639 cpu_manager.go:414] "containerMap: removing container" podUID="5d218ddb-88c3-4646-a31b-cdd5d05dbc1d" containerName="istio-proxy"
Nov 02 02:14:28.455883 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.455861    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d218ddb-88c3-4646-a31b-cdd5d05dbc1d" containerName="istio-proxy"
Nov 02 02:14:28.456143 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.455971    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3" containerName="revoke-keys"
Nov 02 02:14:28.456143 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.455986    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d218ddb-88c3-4646-a31b-cdd5d05dbc1d" containerName="istio-proxy"
Nov 02 02:14:28.456143 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.456002    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d218ddb-88c3-4646-a31b-cdd5d05dbc1d" containerName="istio-proxy"
Nov 02 02:14:28.460720 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.460698    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5064232b-rwcz5"
Nov 02 02:14:28.514839 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.514809    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:14:28.514839 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.514824    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:14:28.566614 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.566579    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5064232b-rwcz5"]
Nov 02 02:14:28.597259 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.597223    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dcecf9b-caa8-4a47-bc54-7e4d1302c843-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5064232b-rwcz5\" (UID: \"0dcecf9b-caa8-4a47-bc54-7e4d1302c843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5064232b-rwcz5"
Nov 02 02:14:28.597461 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.597378    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jntzj\" (UniqueName: \"kubernetes.io/projected/0dcecf9b-caa8-4a47-bc54-7e4d1302c843-kube-api-access-jntzj\") pod \"maas-api-revoke-sub-5064232b-rwcz5\" (UID: \"0dcecf9b-caa8-4a47-bc54-7e4d1302c843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5064232b-rwcz5"
Nov 02 02:14:28.597540 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.597486    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dcecf9b-caa8-4a47-bc54-7e4d1302c843-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5064232b-rwcz5\" (UID: \"0dcecf9b-caa8-4a47-bc54-7e4d1302c843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5064232b-rwcz5"
Nov 02 02:14:28.624442 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.624414    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:14:28.624606 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.624534    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:14:28.676992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.676956    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g"]
Nov 02 02:14:28.698007 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.697979    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jntzj\" (UniqueName: \"kubernetes.io/projected/0dcecf9b-caa8-4a47-bc54-7e4d1302c843-kube-api-access-jntzj\") pod \"maas-api-revoke-sub-5064232b-rwcz5\" (UID: \"0dcecf9b-caa8-4a47-bc54-7e4d1302c843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5064232b-rwcz5"
Nov 02 02:14:28.698144 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.698032    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dcecf9b-caa8-4a47-bc54-7e4d1302c843-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5064232b-rwcz5\" (UID: \"0dcecf9b-caa8-4a47-bc54-7e4d1302c843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5064232b-rwcz5"
Nov 02 02:14:28.698144 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.698078    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dcecf9b-caa8-4a47-bc54-7e4d1302c843-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5064232b-rwcz5\" (UID: \"0dcecf9b-caa8-4a47-bc54-7e4d1302c843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5064232b-rwcz5"
Nov 02 02:14:28.698743 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.698722    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dcecf9b-caa8-4a47-bc54-7e4d1302c843-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5064232b-rwcz5\" (UID: \"0dcecf9b-caa8-4a47-bc54-7e4d1302c843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5064232b-rwcz5"
Nov 02 02:14:28.700743 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.700721    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dcecf9b-caa8-4a47-bc54-7e4d1302c843-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5064232b-rwcz5\" (UID: \"0dcecf9b-caa8-4a47-bc54-7e4d1302c843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5064232b-rwcz5"
Nov 02 02:14:28.759338 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.759259    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jntzj\" (UniqueName: \"kubernetes.io/projected/0dcecf9b-caa8-4a47-bc54-7e4d1302c843-kube-api-access-jntzj\") pod \"maas-api-revoke-sub-5064232b-rwcz5\" (UID: \"0dcecf9b-caa8-4a47-bc54-7e4d1302c843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5064232b-rwcz5"
Nov 02 02:14:28.774088 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:28.774062    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5064232b-rwcz5"
Nov 02 02:14:29.153738 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:29.153707    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5064232b-rwcz5"]
Nov 02 02:14:29.155606 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:14:29.155583    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dcecf9b_caa8_4a47_bc54_7e4d1302c843.slice/crio-d172b4dce1d3b12eb6cbee9165d4876ae9d6966494f9d97376e53bc144ed37ce WatchSource:0}: Error finding container d172b4dce1d3b12eb6cbee9165d4876ae9d6966494f9d97376e53bc144ed37ce: Status 404 returned error can't find the container with id d172b4dce1d3b12eb6cbee9165d4876ae9d6966494f9d97376e53bc144ed37ce
Nov 02 02:14:29.729183 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:29.729092    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5064232b-rwcz5" event={"ID":"0dcecf9b-caa8-4a47-bc54-7e4d1302c843","Type":"ContainerStarted","Data":"260841523bd00af853c5375ae13951c23b538d7994402c050b38d05b170c0ac8"}
Nov 02 02:14:29.729183 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:29.729146    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5064232b-rwcz5" event={"ID":"0dcecf9b-caa8-4a47-bc54-7e4d1302c843","Type":"ContainerStarted","Data":"d172b4dce1d3b12eb6cbee9165d4876ae9d6966494f9d97376e53bc144ed37ce"}
Nov 02 02:14:29.846417 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:29.846367    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5064232b-rwcz5" podStartSLOduration=1.846352032 podStartE2EDuration="1.846352032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:14:29.844478283 +0000 UTC m=+2934.748160395" watchObservedRunningTime="2026-11-02 02:14:29.846352032 +0000 UTC m=+2934.750034143"
Nov 02 02:14:30.736075 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:30.736041    5639 generic.go:356] "Generic (PLEG): container finished" podID="0dcecf9b-caa8-4a47-bc54-7e4d1302c843" containerID="260841523bd00af853c5375ae13951c23b538d7994402c050b38d05b170c0ac8" exitCode=0
Nov 02 02:14:30.736481 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:30.736103    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5064232b-rwcz5" event={"ID":"0dcecf9b-caa8-4a47-bc54-7e4d1302c843","Type":"ContainerDied","Data":"260841523bd00af853c5375ae13951c23b538d7994402c050b38d05b170c0ac8"}
Nov 02 02:14:31.876617 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:31.876587    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5064232b-rwcz5"
Nov 02 02:14:31.932390 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:31.932355    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dcecf9b-caa8-4a47-bc54-7e4d1302c843-kube-api-access-jntzj\" (UniqueName: \"kubernetes.io/projected/0dcecf9b-caa8-4a47-bc54-7e4d1302c843-kube-api-access-jntzj\") pod \"0dcecf9b-caa8-4a47-bc54-7e4d1302c843\" (UID: \"0dcecf9b-caa8-4a47-bc54-7e4d1302c843\") "
Nov 02 02:14:31.932593 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:31.932439    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dcecf9b-caa8-4a47-bc54-7e4d1302c843-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dcecf9b-caa8-4a47-bc54-7e4d1302c843-maas-api-openshift-ca\") pod \"0dcecf9b-caa8-4a47-bc54-7e4d1302c843\" (UID: \"0dcecf9b-caa8-4a47-bc54-7e4d1302c843\") "
Nov 02 02:14:31.932593 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:31.932471    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dcecf9b-caa8-4a47-bc54-7e4d1302c843-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dcecf9b-caa8-4a47-bc54-7e4d1302c843-maas-api-xks-ca\") pod \"0dcecf9b-caa8-4a47-bc54-7e4d1302c843\" (UID: \"0dcecf9b-caa8-4a47-bc54-7e4d1302c843\") "
Nov 02 02:14:31.932752 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:31.932726    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dcecf9b-caa8-4a47-bc54-7e4d1302c843-maas-api-openshift-ca" pod "0dcecf9b-caa8-4a47-bc54-7e4d1302c843" (UID: "0dcecf9b-caa8-4a47-bc54-7e4d1302c843"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:14:31.934812 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:31.934784    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dcecf9b-caa8-4a47-bc54-7e4d1302c843-maas-api-xks-ca" pod "0dcecf9b-caa8-4a47-bc54-7e4d1302c843" (UID: "0dcecf9b-caa8-4a47-bc54-7e4d1302c843"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:14:31.934812 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:31.934788    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dcecf9b-caa8-4a47-bc54-7e4d1302c843-kube-api-access-jntzj" pod "0dcecf9b-caa8-4a47-bc54-7e4d1302c843" (UID: "0dcecf9b-caa8-4a47-bc54-7e4d1302c843"). InnerVolumeSpecName "kube-api-access-jntzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:14:32.034082 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:32.033995    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jntzj\" (UniqueName: \"kubernetes.io/projected/0dcecf9b-caa8-4a47-bc54-7e4d1302c843-kube-api-access-jntzj\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:32.034082 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:32.034027    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dcecf9b-caa8-4a47-bc54-7e4d1302c843-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:32.034082 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:32.034039    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dcecf9b-caa8-4a47-bc54-7e4d1302c843-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:32.749462 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:32.749419    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5064232b-rwcz5"
Nov 02 02:14:32.749665 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:32.749424    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5064232b-rwcz5" event={"ID":"0dcecf9b-caa8-4a47-bc54-7e4d1302c843","Type":"ContainerDied","Data":"d172b4dce1d3b12eb6cbee9165d4876ae9d6966494f9d97376e53bc144ed37ce"}
Nov 02 02:14:32.749665 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:32.749592    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d172b4dce1d3b12eb6cbee9165d4876ae9d6966494f9d97376e53bc144ed37ce"
Nov 02 02:14:33.678219 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:33.678164    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g" podUID="2e96d87f-8aab-450c-8865-b7daec4f91a5" containerName="payload-pre-processing" containerID="cri-o://ca08b6b9822fde2c76f0670856e6b948de5ac92a55b4400aaee83f678c8bf9bb" gracePeriod=25
Nov 02 02:14:36.448993 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.448947    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main" containerID="cri-o://9d8c1830785e6186525eb2eba18cc0e4fa40165f338b420426268500da3f00f7" gracePeriod=45
Nov 02 02:14:36.710611 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.710537    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 02:14:36.774523 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.774488    5639 generic.go:356] "Generic (PLEG): container finished" podID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerID="9d8c1830785e6186525eb2eba18cc0e4fa40165f338b420426268500da3f00f7" exitCode=0
Nov 02 02:14:36.774701 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.774532    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" event={"ID":"fad03279-07f7-4f16-852c-b9c8fb6456dd","Type":"ContainerDied","Data":"9d8c1830785e6186525eb2eba18cc0e4fa40165f338b420426268500da3f00f7"}
Nov 02 02:14:36.774701 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.774568    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92" event={"ID":"fad03279-07f7-4f16-852c-b9c8fb6456dd","Type":"ContainerDied","Data":"ce4d435461dadadbeb5a42e21d301f67635957bbfe0e408d7b5728cdc38650ce"}
Nov 02 02:14:36.774701 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.774589    5639 scope.go:122] "RemoveContainer" containerID="9d8c1830785e6186525eb2eba18cc0e4fa40165f338b420426268500da3f00f7"
Nov 02 02:14:36.774701 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.774533    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"
Nov 02 02:14:36.779518 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.779495    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fad03279-07f7-4f16-852c-b9c8fb6456dd-kube-api-access-k5bl4\" (UniqueName: \"kubernetes.io/projected/fad03279-07f7-4f16-852c-b9c8fb6456dd-kube-api-access-k5bl4\") pod \"fad03279-07f7-4f16-852c-b9c8fb6456dd\" (UID: \"fad03279-07f7-4f16-852c-b9c8fb6456dd\") "
Nov 02 02:14:36.779672 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.779536    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-model-cache\") pod \"fad03279-07f7-4f16-852c-b9c8fb6456dd\" (UID: \"fad03279-07f7-4f16-852c-b9c8fb6456dd\") "
Nov 02 02:14:36.779672 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.779585    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-tmp-dir\") pod \"fad03279-07f7-4f16-852c-b9c8fb6456dd\" (UID: \"fad03279-07f7-4f16-852c-b9c8fb6456dd\") "
Nov 02 02:14:36.779672 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.779629    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-home\" (UniqueName: \"kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-home\") pod \"fad03279-07f7-4f16-852c-b9c8fb6456dd\" (UID: \"fad03279-07f7-4f16-852c-b9c8fb6456dd\") "
Nov 02 02:14:36.779672 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.779659    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fad03279-07f7-4f16-852c-b9c8fb6456dd-tls-certs\" (UniqueName: \"kubernetes.io/secret/fad03279-07f7-4f16-852c-b9c8fb6456dd-tls-certs\") pod \"fad03279-07f7-4f16-852c-b9c8fb6456dd\" (UID: \"fad03279-07f7-4f16-852c-b9c8fb6456dd\") "
Nov 02 02:14:36.779917 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.779763    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-dshm\") pod \"fad03279-07f7-4f16-852c-b9c8fb6456dd\" (UID: \"fad03279-07f7-4f16-852c-b9c8fb6456dd\") "
Nov 02 02:14:36.779917 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.779893    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-home" pod "fad03279-07f7-4f16-852c-b9c8fb6456dd" (UID: "fad03279-07f7-4f16-852c-b9c8fb6456dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:36.779917 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.779904    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-model-cache" pod "fad03279-07f7-4f16-852c-b9c8fb6456dd" (UID: "fad03279-07f7-4f16-852c-b9c8fb6456dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:36.780073 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.779931    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-tmp-dir" pod "fad03279-07f7-4f16-852c-b9c8fb6456dd" (UID: "fad03279-07f7-4f16-852c-b9c8fb6456dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:36.780417 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.780393    5639 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-model-cache\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:36.780417 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.780417    5639 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-tmp-dir\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:36.780616 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.780426    5639 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-home\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:36.782029 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.781989    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad03279-07f7-4f16-852c-b9c8fb6456dd-kube-api-access-k5bl4" pod "fad03279-07f7-4f16-852c-b9c8fb6456dd" (UID: "fad03279-07f7-4f16-852c-b9c8fb6456dd"). InnerVolumeSpecName "kube-api-access-k5bl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:14:36.782118 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.782024    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fad03279-07f7-4f16-852c-b9c8fb6456dd-tls-certs" pod "fad03279-07f7-4f16-852c-b9c8fb6456dd" (UID: "fad03279-07f7-4f16-852c-b9c8fb6456dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:14:36.782224 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.782206    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-dshm" pod "fad03279-07f7-4f16-852c-b9c8fb6456dd" (UID: "fad03279-07f7-4f16-852c-b9c8fb6456dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:36.786075 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.786046    5639 scope.go:122] "RemoveContainer" containerID="9d8c1830785e6186525eb2eba18cc0e4fa40165f338b420426268500da3f00f7"
Nov 02 02:14:36.786374 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:14:36.786341    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d8c1830785e6186525eb2eba18cc0e4fa40165f338b420426268500da3f00f7\": container with ID starting with 9d8c1830785e6186525eb2eba18cc0e4fa40165f338b420426268500da3f00f7 not found: ID does not exist" containerID="9d8c1830785e6186525eb2eba18cc0e4fa40165f338b420426268500da3f00f7"
Nov 02 02:14:36.786458 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.786382    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d8c1830785e6186525eb2eba18cc0e4fa40165f338b420426268500da3f00f7"} err="failed to get container status \"9d8c1830785e6186525eb2eba18cc0e4fa40165f338b420426268500da3f00f7\": rpc error: code = NotFound desc = could not find container \"9d8c1830785e6186525eb2eba18cc0e4fa40165f338b420426268500da3f00f7\": container with ID starting with 9d8c1830785e6186525eb2eba18cc0e4fa40165f338b420426268500da3f00f7 not found: ID does not exist"
Nov 02 02:14:36.881961 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.881914    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5bl4\" (UniqueName: \"kubernetes.io/projected/fad03279-07f7-4f16-852c-b9c8fb6456dd-kube-api-access-k5bl4\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:36.881961 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.881955    5639 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fad03279-07f7-4f16-852c-b9c8fb6456dd-tls-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:36.882217 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:36.881973    5639 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fad03279-07f7-4f16-852c-b9c8fb6456dd-dshm\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:37.193525 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:37.193490    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"]
Nov 02 02:14:37.197837 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:37.197809    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e06ea2/distinct-2-w5-e06ea2-kserve-6dbd7ff79b-x4n92"]
Nov 02 02:14:37.217562 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:37.217535    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" path="/var/lib/kubelet/pods/fad03279-07f7-4f16-852c-b9c8fb6456dd/volumes"
Nov 02 02:14:39.094011 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:39.093971    5639 scope.go:122] "RemoveContainer" containerID="a41e8a2c8550d0ef2345a0c2ea47abb0d884247da8eb4f0ded8970dfbe3b37c7"
Nov 02 02:14:39.506220 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:39.506132    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g" podUID="2e96d87f-8aab-450c-8865-b7daec4f91a5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:14:45.026304 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:45.026243    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:14:45.226419 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:45.226384    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr"]
Nov 02 02:14:45.276126 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:45.276088    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e06ea2-zqxzr"]
Nov 02 02:14:47.217401 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:47.217357    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3" path="/var/lib/kubelet/pods/6359e29f-3c9d-4192-8c6e-7b1a13b4a2a3/volumes"
Nov 02 02:14:49.506062 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:49.506025    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g" podUID="2e96d87f-8aab-450c-8865-b7daec4f91a5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:14:53.873274 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:53.873230    5639 generic.go:356] "Generic (PLEG): container finished" podID="2e96d87f-8aab-450c-8865-b7daec4f91a5" containerID="ca08b6b9822fde2c76f0670856e6b948de5ac92a55b4400aaee83f678c8bf9bb" exitCode=0
Nov 02 02:14:53.873821 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:53.873319    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g" event={"ID":"2e96d87f-8aab-450c-8865-b7daec4f91a5","Type":"ContainerDied","Data":"ca08b6b9822fde2c76f0670856e6b948de5ac92a55b4400aaee83f678c8bf9bb"}
Nov 02 02:14:53.933679 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:53.933646    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g"
Nov 02 02:14:54.045505 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:54.045408    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e96d87f-8aab-450c-8865-b7daec4f91a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e96d87f-8aab-450c-8865-b7daec4f91a5-plugins-config-volume\") pod \"2e96d87f-8aab-450c-8865-b7daec4f91a5\" (UID: \"2e96d87f-8aab-450c-8865-b7daec4f91a5\") "
Nov 02 02:14:54.045678 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:54.045567    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e96d87f-8aab-450c-8865-b7daec4f91a5-kube-api-access-79vjc\" (UniqueName: \"kubernetes.io/projected/2e96d87f-8aab-450c-8865-b7daec4f91a5-kube-api-access-79vjc\") pod \"2e96d87f-8aab-450c-8865-b7daec4f91a5\" (UID: \"2e96d87f-8aab-450c-8865-b7daec4f91a5\") "
Nov 02 02:14:54.045866 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:54.045841    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e96d87f-8aab-450c-8865-b7daec4f91a5-plugins-config-volume" pod "2e96d87f-8aab-450c-8865-b7daec4f91a5" (UID: "2e96d87f-8aab-450c-8865-b7daec4f91a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:14:54.047839 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:54.047811    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e96d87f-8aab-450c-8865-b7daec4f91a5-kube-api-access-79vjc" pod "2e96d87f-8aab-450c-8865-b7daec4f91a5" (UID: "2e96d87f-8aab-450c-8865-b7daec4f91a5"). InnerVolumeSpecName "kube-api-access-79vjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:14:54.146778 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:54.146720    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e96d87f-8aab-450c-8865-b7daec4f91a5-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:54.146778 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:54.146771    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79vjc\" (UniqueName: \"kubernetes.io/projected/2e96d87f-8aab-450c-8865-b7daec4f91a5-kube-api-access-79vjc\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:54.880040 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:54.880012    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g"
Nov 02 02:14:54.880040 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:54.880018    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g" event={"ID":"2e96d87f-8aab-450c-8865-b7daec4f91a5","Type":"ContainerDied","Data":"cd92539acb0d8449ca7e1e3803d7d5a7a3960469ac0bf29160df3a5564b67781"}
Nov 02 02:14:54.880561 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:54.880087    5639 scope.go:122] "RemoveContainer" containerID="ca08b6b9822fde2c76f0670856e6b948de5ac92a55b4400aaee83f678c8bf9bb"
Nov 02 02:14:55.044130 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:55.044093    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g"]
Nov 02 02:14:55.093463 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:55.093433    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e06ea2-765b747687-vvd6g"]
Nov 02 02:14:55.216969 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:55.216897    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e96d87f-8aab-450c-8865-b7daec4f91a5" path="/var/lib/kubelet/pods/2e96d87f-8aab-450c-8865-b7daec4f91a5/volumes"
Nov 02 02:14:57.420126 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:14:57.420082    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:14:57.434236 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:57.434204    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65b55dd8bb-jrkxd"]
Nov 02 02:14:57.434471 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:57.434446    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-65b55dd8bb-jrkxd" podUID="71d1816d-03d5-4be7-9b26-69c6da974752" containerName="manager" containerID="cri-o://d428e16296d41f7ee3edb8c00b14c911250646a5679d1d8375bf143f620d5beb" gracePeriod=30
Nov 02 02:14:57.687299 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:57.687253    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65b55dd8bb-jrkxd"
Nov 02 02:14:57.777812 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:57.777776    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71d1816d-03d5-4be7-9b26-69c6da974752-webhook-cert\" (UniqueName: \"kubernetes.io/secret/71d1816d-03d5-4be7-9b26-69c6da974752-webhook-cert\") pod \"71d1816d-03d5-4be7-9b26-69c6da974752\" (UID: \"71d1816d-03d5-4be7-9b26-69c6da974752\") "
Nov 02 02:14:57.778017 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:57.777842    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d1816d-03d5-4be7-9b26-69c6da974752-kube-api-access-29z9w\" (UniqueName: \"kubernetes.io/projected/71d1816d-03d5-4be7-9b26-69c6da974752-kube-api-access-29z9w\") pod \"71d1816d-03d5-4be7-9b26-69c6da974752\" (UID: \"71d1816d-03d5-4be7-9b26-69c6da974752\") "
Nov 02 02:14:57.778017 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:57.777889    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71d1816d-03d5-4be7-9b26-69c6da974752-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d1816d-03d5-4be7-9b26-69c6da974752-metrics-certs\") pod \"71d1816d-03d5-4be7-9b26-69c6da974752\" (UID: \"71d1816d-03d5-4be7-9b26-69c6da974752\") "
Nov 02 02:14:57.780165 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:57.780130    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71d1816d-03d5-4be7-9b26-69c6da974752-webhook-cert" pod "71d1816d-03d5-4be7-9b26-69c6da974752" (UID: "71d1816d-03d5-4be7-9b26-69c6da974752"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:14:57.780165 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:57.780155    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71d1816d-03d5-4be7-9b26-69c6da974752-metrics-certs" pod "71d1816d-03d5-4be7-9b26-69c6da974752" (UID: "71d1816d-03d5-4be7-9b26-69c6da974752"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:14:57.780413 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:57.780164    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d1816d-03d5-4be7-9b26-69c6da974752-kube-api-access-29z9w" pod "71d1816d-03d5-4be7-9b26-69c6da974752" (UID: "71d1816d-03d5-4be7-9b26-69c6da974752"). InnerVolumeSpecName "kube-api-access-29z9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:14:57.879180 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:57.879142    5639 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71d1816d-03d5-4be7-9b26-69c6da974752-webhook-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:57.879180 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:57.879178    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29z9w\" (UniqueName: \"kubernetes.io/projected/71d1816d-03d5-4be7-9b26-69c6da974752-kube-api-access-29z9w\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:57.879448 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:57.879194    5639 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d1816d-03d5-4be7-9b26-69c6da974752-metrics-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:57.896638 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:57.896609    5639 generic.go:356] "Generic (PLEG): container finished" podID="71d1816d-03d5-4be7-9b26-69c6da974752" containerID="d428e16296d41f7ee3edb8c00b14c911250646a5679d1d8375bf143f620d5beb" exitCode=0
Nov 02 02:14:57.896809 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:57.896657    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65b55dd8bb-jrkxd"
Nov 02 02:14:57.896809 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:57.896707    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65b55dd8bb-jrkxd" event={"ID":"71d1816d-03d5-4be7-9b26-69c6da974752","Type":"ContainerDied","Data":"d428e16296d41f7ee3edb8c00b14c911250646a5679d1d8375bf143f620d5beb"}
Nov 02 02:14:57.896809 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:57.896764    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65b55dd8bb-jrkxd" event={"ID":"71d1816d-03d5-4be7-9b26-69c6da974752","Type":"ContainerDied","Data":"de14d6264f752819ec71bfa097ad3c0dd98974d37b06e5f693dd6da70196bbea"}
Nov 02 02:14:57.896809 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:57.896792    5639 scope.go:122] "RemoveContainer" containerID="d428e16296d41f7ee3edb8c00b14c911250646a5679d1d8375bf143f620d5beb"
Nov 02 02:14:57.907249 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:57.907228    5639 scope.go:122] "RemoveContainer" containerID="d428e16296d41f7ee3edb8c00b14c911250646a5679d1d8375bf143f620d5beb"
Nov 02 02:14:57.907539 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:14:57.907514    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d428e16296d41f7ee3edb8c00b14c911250646a5679d1d8375bf143f620d5beb\": container with ID starting with d428e16296d41f7ee3edb8c00b14c911250646a5679d1d8375bf143f620d5beb not found: ID does not exist" containerID="d428e16296d41f7ee3edb8c00b14c911250646a5679d1d8375bf143f620d5beb"
Nov 02 02:14:57.907631 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:57.907549    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d428e16296d41f7ee3edb8c00b14c911250646a5679d1d8375bf143f620d5beb"} err="failed to get container status \"d428e16296d41f7ee3edb8c00b14c911250646a5679d1d8375bf143f620d5beb\": rpc error: code = NotFound desc = could not find container \"d428e16296d41f7ee3edb8c00b14c911250646a5679d1d8375bf143f620d5beb\": container with ID starting with d428e16296d41f7ee3edb8c00b14c911250646a5679d1d8375bf143f620d5beb not found: ID does not exist"
Nov 02 02:14:58.059850 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:58.059762    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65b55dd8bb-jrkxd"]
Nov 02 02:14:58.063723 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:58.063694    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-65b55dd8bb-jrkxd"]
Nov 02 02:14:58.614632 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:58.614594    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:14:58.887151 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:58.887062    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65b55dd8bb-482b8"]
Nov 02 02:14:58.888078 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:58.888052    5639 cpu_manager.go:414] "containerMap: removing container" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main"
Nov 02 02:14:58.888078 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:58.888077    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main"
Nov 02 02:14:58.888298 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:58.888106    5639 cpu_manager.go:414] "containerMap: removing container" podUID="2e96d87f-8aab-450c-8865-b7daec4f91a5" containerName="payload-pre-processing"
Nov 02 02:14:58.888298 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:58.888114    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e96d87f-8aab-450c-8865-b7daec4f91a5" containerName="payload-pre-processing"
Nov 02 02:14:58.888298 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:58.888141    5639 cpu_manager.go:414] "containerMap: removing container" podUID="0dcecf9b-caa8-4a47-bc54-7e4d1302c843" containerName="revoke-keys"
Nov 02 02:14:58.888298 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:58.888151    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dcecf9b-caa8-4a47-bc54-7e4d1302c843" containerName="revoke-keys"
Nov 02 02:14:58.888298 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:58.888181    5639 cpu_manager.go:414] "containerMap: removing container" podUID="71d1816d-03d5-4be7-9b26-69c6da974752" containerName="manager"
Nov 02 02:14:58.888298 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:58.888189    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d1816d-03d5-4be7-9b26-69c6da974752" containerName="manager"
Nov 02 02:14:58.888564 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:58.888399    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="fad03279-07f7-4f16-852c-b9c8fb6456dd" containerName="main"
Nov 02 02:14:58.888564 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:58.888412    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e96d87f-8aab-450c-8865-b7daec4f91a5" containerName="payload-pre-processing"
Nov 02 02:14:58.888564 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:58.888430    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dcecf9b-caa8-4a47-bc54-7e4d1302c843" containerName="revoke-keys"
Nov 02 02:14:58.888564 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:58.888442    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d1816d-03d5-4be7-9b26-69c6da974752" containerName="manager"
Nov 02 02:14:58.893835 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:58.893805    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65b55dd8bb-482b8"
Nov 02 02:14:58.947246 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:58.947212    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 02:14:58.947463 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:58.947415    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 02:14:58.949224 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:58.949194    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65b55dd8bb-482b8"]
Nov 02 02:14:58.995730 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:58.995698    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ghnpp\""
Nov 02 02:14:58.995730 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:58.995716    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 02:14:58.997083 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:58.997057    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cfc5266-30c3-4306-afbe-d1ae5c102df5-metrics-certs\") pod \"maas-controller-65b55dd8bb-482b8\" (UID: \"3cfc5266-30c3-4306-afbe-d1ae5c102df5\") " pod="opendatahub/maas-controller-65b55dd8bb-482b8"
Nov 02 02:14:58.997205 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:58.997097    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3cfc5266-30c3-4306-afbe-d1ae5c102df5-webhook-cert\") pod \"maas-controller-65b55dd8bb-482b8\" (UID: \"3cfc5266-30c3-4306-afbe-d1ae5c102df5\") " pod="opendatahub/maas-controller-65b55dd8bb-482b8"
Nov 02 02:14:58.997338 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:58.997313    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn92h\" (UniqueName: \"kubernetes.io/projected/3cfc5266-30c3-4306-afbe-d1ae5c102df5-kube-api-access-kn92h\") pod \"maas-controller-65b55dd8bb-482b8\" (UID: \"3cfc5266-30c3-4306-afbe-d1ae5c102df5\") " pod="opendatahub/maas-controller-65b55dd8bb-482b8"
Nov 02 02:14:59.098320 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:59.098263    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn92h\" (UniqueName: \"kubernetes.io/projected/3cfc5266-30c3-4306-afbe-d1ae5c102df5-kube-api-access-kn92h\") pod \"maas-controller-65b55dd8bb-482b8\" (UID: \"3cfc5266-30c3-4306-afbe-d1ae5c102df5\") " pod="opendatahub/maas-controller-65b55dd8bb-482b8"
Nov 02 02:14:59.098505 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:59.098364    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cfc5266-30c3-4306-afbe-d1ae5c102df5-metrics-certs\") pod \"maas-controller-65b55dd8bb-482b8\" (UID: \"3cfc5266-30c3-4306-afbe-d1ae5c102df5\") " pod="opendatahub/maas-controller-65b55dd8bb-482b8"
Nov 02 02:14:59.098505 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:59.098387    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3cfc5266-30c3-4306-afbe-d1ae5c102df5-webhook-cert\") pod \"maas-controller-65b55dd8bb-482b8\" (UID: \"3cfc5266-30c3-4306-afbe-d1ae5c102df5\") " pod="opendatahub/maas-controller-65b55dd8bb-482b8"
Nov 02 02:14:59.101057 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:59.101028    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cfc5266-30c3-4306-afbe-d1ae5c102df5-metrics-certs\") pod \"maas-controller-65b55dd8bb-482b8\" (UID: \"3cfc5266-30c3-4306-afbe-d1ae5c102df5\") " pod="opendatahub/maas-controller-65b55dd8bb-482b8"
Nov 02 02:14:59.101212 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:59.101060    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3cfc5266-30c3-4306-afbe-d1ae5c102df5-webhook-cert\") pod \"maas-controller-65b55dd8bb-482b8\" (UID: \"3cfc5266-30c3-4306-afbe-d1ae5c102df5\") " pod="opendatahub/maas-controller-65b55dd8bb-482b8"
Nov 02 02:14:59.159448 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:59.159366    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn92h\" (UniqueName: \"kubernetes.io/projected/3cfc5266-30c3-4306-afbe-d1ae5c102df5-kube-api-access-kn92h\") pod \"maas-controller-65b55dd8bb-482b8\" (UID: \"3cfc5266-30c3-4306-afbe-d1ae5c102df5\") " pod="opendatahub/maas-controller-65b55dd8bb-482b8"
Nov 02 02:14:59.209700 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:59.209662    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65b55dd8bb-482b8"
Nov 02 02:14:59.218087 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:59.218046    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71d1816d-03d5-4be7-9b26-69c6da974752" path="/var/lib/kubelet/pods/71d1816d-03d5-4be7-9b26-69c6da974752/volumes"
Nov 02 02:14:59.587543 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:59.587505    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65b55dd8bb-482b8"]
Nov 02 02:14:59.619116 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:14:59.619082    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cfc5266_30c3_4306_afbe_d1ae5c102df5.slice/crio-7ef1cd530928566a911e9d3db02bf758dbd695711c2ab0d01fbb71bcea2c3141 WatchSource:0}: Error finding container 7ef1cd530928566a911e9d3db02bf758dbd695711c2ab0d01fbb71bcea2c3141: Status 404 returned error can't find the container with id 7ef1cd530928566a911e9d3db02bf758dbd695711c2ab0d01fbb71bcea2c3141
Nov 02 02:14:59.620521 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:59.620504    5639 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 02:14:59.912434 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:14:59.912326    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65b55dd8bb-482b8" event={"ID":"3cfc5266-30c3-4306-afbe-d1ae5c102df5","Type":"ContainerStarted","Data":"7ef1cd530928566a911e9d3db02bf758dbd695711c2ab0d01fbb71bcea2c3141"}
Nov 02 02:15:00.169856 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:00.169754    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893095-86gp6"]
Nov 02 02:15:00.175482 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:00.175455    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893095-86gp6"
Nov 02 02:15:00.208886 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:00.208838    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn76z\" (UniqueName: \"kubernetes.io/projected/03aac5c9-0853-4f79-bc5d-f785b322ce0c-kube-api-access-rn76z\") pod \"maas-api-key-cleanup-29893095-86gp6\" (UID: \"03aac5c9-0853-4f79-bc5d-f785b322ce0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893095-86gp6"
Nov 02 02:15:00.209084 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:00.208981    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03aac5c9-0853-4f79-bc5d-f785b322ce0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893095-86gp6\" (UID: \"03aac5c9-0853-4f79-bc5d-f785b322ce0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893095-86gp6"
Nov 02 02:15:00.228179 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:00.228141    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:15:00.228179 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:00.228159    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:15:00.228433 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:00.228206    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:15:00.230198 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:00.230172    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893095-86gp6"]
Nov 02 02:15:00.309853 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:00.309814    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03aac5c9-0853-4f79-bc5d-f785b322ce0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893095-86gp6\" (UID: \"03aac5c9-0853-4f79-bc5d-f785b322ce0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893095-86gp6"
Nov 02 02:15:00.310084 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:00.309953    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn76z\" (UniqueName: \"kubernetes.io/projected/03aac5c9-0853-4f79-bc5d-f785b322ce0c-kube-api-access-rn76z\") pod \"maas-api-key-cleanup-29893095-86gp6\" (UID: \"03aac5c9-0853-4f79-bc5d-f785b322ce0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893095-86gp6"
Nov 02 02:15:00.310602 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:00.310568    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03aac5c9-0853-4f79-bc5d-f785b322ce0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893095-86gp6\" (UID: \"03aac5c9-0853-4f79-bc5d-f785b322ce0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893095-86gp6"
Nov 02 02:15:00.371546 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:00.371509    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn76z\" (UniqueName: \"kubernetes.io/projected/03aac5c9-0853-4f79-bc5d-f785b322ce0c-kube-api-access-rn76z\") pod \"maas-api-key-cleanup-29893095-86gp6\" (UID: \"03aac5c9-0853-4f79-bc5d-f785b322ce0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893095-86gp6"
Nov 02 02:15:00.493800 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:00.493704    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893095-86gp6"
Nov 02 02:15:00.874830 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:00.874802    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893095-86gp6"]
Nov 02 02:15:00.878429 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:15:00.878401    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03aac5c9_0853_4f79_bc5d_f785b322ce0c.slice/crio-0e912ab52109d468c379e03dec4352089f30b23c13926536c3ad376eb49ce2b2 WatchSource:0}: Error finding container 0e912ab52109d468c379e03dec4352089f30b23c13926536c3ad376eb49ce2b2: Status 404 returned error can't find the container with id 0e912ab52109d468c379e03dec4352089f30b23c13926536c3ad376eb49ce2b2
Nov 02 02:15:00.920062 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:00.920025    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65b55dd8bb-482b8" event={"ID":"3cfc5266-30c3-4306-afbe-d1ae5c102df5","Type":"ContainerStarted","Data":"cc16f44a04ec59e1cd2007248be15da4382c72dc85e02ae50bab520ce6c9ae05"}
Nov 02 02:15:00.920243 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:00.920216    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65b55dd8bb-482b8"
Nov 02 02:15:00.921419 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:00.921398    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893095-86gp6" event={"ID":"03aac5c9-0853-4f79-bc5d-f785b322ce0c","Type":"ContainerStarted","Data":"0e912ab52109d468c379e03dec4352089f30b23c13926536c3ad376eb49ce2b2"}
Nov 02 02:15:01.038552 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:01.038494    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-65b55dd8bb-482b8" podStartSLOduration=2.119159271 podStartE2EDuration="3.038479401s" totalImagesPullingTime="919.32013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:14:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:15:01.036502766 +0000 UTC m=+2965.940184880" watchObservedRunningTime="2026-11-02 02:15:01.038479401 +0000 UTC m=+2965.942161554"
Nov 02 02:15:01.929360 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:01.929242    5639 generic.go:356] "Generic (PLEG): container finished" podID="03aac5c9-0853-4f79-bc5d-f785b322ce0c" containerID="f9747df1f0954863e587080e4cb3136d2737deab34209efdad2815f74c158451" exitCode=60
Nov 02 02:15:01.929360 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:01.929310    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893095-86gp6" event={"ID":"03aac5c9-0853-4f79-bc5d-f785b322ce0c","Type":"ContainerDied","Data":"f9747df1f0954863e587080e4cb3136d2737deab34209efdad2815f74c158451"}
Nov 02 02:15:01.929939 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:01.929620    5639 scope.go:122] "RemoveContainer" containerID="f9747df1f0954863e587080e4cb3136d2737deab34209efdad2815f74c158451"
Nov 02 02:15:02.936864 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:02.936750    5639 generic.go:356] "Generic (PLEG): container finished" podID="03aac5c9-0853-4f79-bc5d-f785b322ce0c" containerID="3d3d922c7d772494773f1807664a9fae8f250290ef23052954b44b49cab631d2" exitCode=60
Nov 02 02:15:02.936864 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:02.936785    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893095-86gp6" event={"ID":"03aac5c9-0853-4f79-bc5d-f785b322ce0c","Type":"ContainerDied","Data":"3d3d922c7d772494773f1807664a9fae8f250290ef23052954b44b49cab631d2"}
Nov 02 02:15:02.936864 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:02.936850    5639 scope.go:122] "RemoveContainer" containerID="f9747df1f0954863e587080e4cb3136d2737deab34209efdad2815f74c158451"
Nov 02 02:15:02.937427 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:02.937122    5639 scope.go:122] "RemoveContainer" containerID="3d3d922c7d772494773f1807664a9fae8f250290ef23052954b44b49cab631d2"
Nov 02 02:15:02.937491 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:15:02.937424    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893095-86gp6_odh-ai-gateway-infra(03aac5c9-0853-4f79-bc5d-f785b322ce0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893095-86gp6" podUID="03aac5c9-0853-4f79-bc5d-f785b322ce0c"
Nov 02 02:15:03.943899 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:03.943872    5639 scope.go:122] "RemoveContainer" containerID="3d3d922c7d772494773f1807664a9fae8f250290ef23052954b44b49cab631d2"
Nov 02 02:15:03.944365 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:15:03.944078    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893095-86gp6_odh-ai-gateway-infra(03aac5c9-0853-4f79-bc5d-f785b322ce0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893095-86gp6" podUID="03aac5c9-0853-4f79-bc5d-f785b322ce0c"
Nov 02 02:15:09.106012 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.105978    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"]
Nov 02 02:15:09.109683 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.109663    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.194753 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.194718    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.194969 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.194769    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.194969 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.194839    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.194969 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.194876    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjzj8\" (UniqueName: \"kubernetes.io/projected/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-kube-api-access-mjzj8\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.194969 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.194956    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.195222 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.194995    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.195222 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.195088    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.195222 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.195161    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.195404 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.195233    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.225544 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.225510    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"]
Nov 02 02:15:09.296154 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.296114    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.296410 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.296170    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.296410 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.296215    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.296410 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.296239    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.296410 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.296312    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.296410 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.296344    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.296410 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.296373    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.296410 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.296395    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjzj8\" (UniqueName: \"kubernetes.io/projected/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-kube-api-access-mjzj8\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.296779 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.296429    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.296951 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.296913    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.297097 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.296952    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.297097 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.296966    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.297097 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.296976    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.297097 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.297082    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.298943 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.298914    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.299156 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.299139    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.357979 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.357902    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjzj8\" (UniqueName: \"kubernetes.io/projected/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-kube-api-access-mjzj8\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.357979 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.357918    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70ec0d83-9e6c-4e20-82b6-2f09909fc85f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qnpm7\" (UID: \"70ec0d83-9e6c-4e20-82b6-2f09909fc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.422859 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.422820    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:09.848975 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.848947    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"]
Nov 02 02:15:09.851991 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:15:09.851962    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ec0d83_9e6c_4e20_82b6_2f09909fc85f.slice/crio-8750e44ec2e78e11050fc42dcbb9e41389c5c097023df42c71aa4e4fd4d758b5 WatchSource:0}: Error finding container 8750e44ec2e78e11050fc42dcbb9e41389c5c097023df42c71aa4e4fd4d758b5: Status 404 returned error can't find the container with id 8750e44ec2e78e11050fc42dcbb9e41389c5c097023df42c71aa4e4fd4d758b5
Nov 02 02:15:09.854210 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.854175    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:15:09.854308 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.854243    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:15:09.854308 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.854272    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:15:09.977509 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.977466    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7" event={"ID":"70ec0d83-9e6c-4e20-82b6-2f09909fc85f","Type":"ContainerStarted","Data":"34dcf9da5bcde9f2da7cf715ffbfcd4b9023de8162e6ea021eb659559302cd95"}
Nov 02 02:15:09.977715 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:09.977536    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7" event={"ID":"70ec0d83-9e6c-4e20-82b6-2f09909fc85f","Type":"ContainerStarted","Data":"8750e44ec2e78e11050fc42dcbb9e41389c5c097023df42c71aa4e4fd4d758b5"}
Nov 02 02:15:10.097162 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:10.097103    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7" podStartSLOduration=1.097083971 podStartE2EDuration="1.097083971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:15:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:15:10.095896423 +0000 UTC m=+2974.999578536" watchObservedRunningTime="2026-11-02 02:15:10.097083971 +0000 UTC m=+2975.000766095"
Nov 02 02:15:10.423369 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:10.423331    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:10.424924 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:10.424900    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body=
Nov 02 02:15:10.425046 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:10.424959    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7" podUID="70ec0d83-9e6c-4e20-82b6-2f09909fc85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused"
Nov 02 02:15:11.423769 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:11.423731    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body=
Nov 02 02:15:11.424198 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:11.423782    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7" podUID="70ec0d83-9e6c-4e20-82b6-2f09909fc85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused"
Nov 02 02:15:11.932008 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:11.931974    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65b55dd8bb-482b8"
Nov 02 02:15:12.424258 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:12.424222    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body=
Nov 02 02:15:12.424766 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:12.424273    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7" podUID="70ec0d83-9e6c-4e20-82b6-2f09909fc85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused"
Nov 02 02:15:13.423463 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:13.423420    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body=
Nov 02 02:15:13.423679 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:13.423494    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7" podUID="70ec0d83-9e6c-4e20-82b6-2f09909fc85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused"
Nov 02 02:15:14.423694 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:14.423660    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body=
Nov 02 02:15:14.424102 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:14.423712    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7" podUID="70ec0d83-9e6c-4e20-82b6-2f09909fc85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused"
Nov 02 02:15:15.423548 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:15.423454    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body=
Nov 02 02:15:15.423548 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:15.423533    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7" podUID="70ec0d83-9e6c-4e20-82b6-2f09909fc85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused"
Nov 02 02:15:16.211215 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:16.211175    5639 scope.go:122] "RemoveContainer" containerID="3d3d922c7d772494773f1807664a9fae8f250290ef23052954b44b49cab631d2"
Nov 02 02:15:16.423684 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:16.423646    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body=
Nov 02 02:15:16.423858 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:16.423710    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7" podUID="70ec0d83-9e6c-4e20-82b6-2f09909fc85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused"
Nov 02 02:15:17.022801 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:17.022757    5639 generic.go:356] "Generic (PLEG): container finished" podID="03aac5c9-0853-4f79-bc5d-f785b322ce0c" containerID="4e8042cb8485eeb69d17be66cc4511a6b0078b8300d4ae1621c80dfc213f14e0" exitCode=60
Nov 02 02:15:17.022987 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:17.022831    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893095-86gp6" event={"ID":"03aac5c9-0853-4f79-bc5d-f785b322ce0c","Type":"ContainerDied","Data":"4e8042cb8485eeb69d17be66cc4511a6b0078b8300d4ae1621c80dfc213f14e0"}
Nov 02 02:15:17.022987 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:17.022942    5639 scope.go:122] "RemoveContainer" containerID="3d3d922c7d772494773f1807664a9fae8f250290ef23052954b44b49cab631d2"
Nov 02 02:15:17.023389 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:17.023370    5639 scope.go:122] "RemoveContainer" containerID="4e8042cb8485eeb69d17be66cc4511a6b0078b8300d4ae1621c80dfc213f14e0"
Nov 02 02:15:17.023685 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:15:17.023651    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893095-86gp6_odh-ai-gateway-infra(03aac5c9-0853-4f79-bc5d-f785b322ce0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893095-86gp6" podUID="03aac5c9-0853-4f79-bc5d-f785b322ce0c"
Nov 02 02:15:18.054531 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:18.054496    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:18.055017 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:18.054878    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:18.055754 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:18.055737    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7"
Nov 02 02:15:18.158073 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:18.158045    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893095-86gp6"]
Nov 02 02:15:18.292101 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:18.292077    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893095-86gp6"
Nov 02 02:15:18.394356 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:18.394319    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03aac5c9-0853-4f79-bc5d-f785b322ce0c-kube-api-access-rn76z\" (UniqueName: \"kubernetes.io/projected/03aac5c9-0853-4f79-bc5d-f785b322ce0c-kube-api-access-rn76z\") pod \"03aac5c9-0853-4f79-bc5d-f785b322ce0c\" (UID: \"03aac5c9-0853-4f79-bc5d-f785b322ce0c\") "
Nov 02 02:15:18.394556 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:18.394497    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03aac5c9-0853-4f79-bc5d-f785b322ce0c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03aac5c9-0853-4f79-bc5d-f785b322ce0c-maas-api-service-ca\") pod \"03aac5c9-0853-4f79-bc5d-f785b322ce0c\" (UID: \"03aac5c9-0853-4f79-bc5d-f785b322ce0c\") "
Nov 02 02:15:18.394935 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:18.394907    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03aac5c9-0853-4f79-bc5d-f785b322ce0c-maas-api-service-ca" pod "03aac5c9-0853-4f79-bc5d-f785b322ce0c" (UID: "03aac5c9-0853-4f79-bc5d-f785b322ce0c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:15:18.396549 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:18.396519    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03aac5c9-0853-4f79-bc5d-f785b322ce0c-kube-api-access-rn76z" pod "03aac5c9-0853-4f79-bc5d-f785b322ce0c" (UID: "03aac5c9-0853-4f79-bc5d-f785b322ce0c"). InnerVolumeSpecName "kube-api-access-rn76z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:15:18.495345 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:18.495315    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03aac5c9-0853-4f79-bc5d-f785b322ce0c-maas-api-service-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:18.495345 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:18.495342    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn76z\" (UniqueName: \"kubernetes.io/projected/03aac5c9-0853-4f79-bc5d-f785b322ce0c-kube-api-access-rn76z\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:19.037571 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:19.037539    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893095-86gp6"
Nov 02 02:15:19.037571 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:19.037564    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893095-86gp6" event={"ID":"03aac5c9-0853-4f79-bc5d-f785b322ce0c","Type":"ContainerDied","Data":"0e912ab52109d468c379e03dec4352089f30b23c13926536c3ad376eb49ce2b2"}
Nov 02 02:15:19.037851 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:19.037626    5639 scope.go:122] "RemoveContainer" containerID="4e8042cb8485eeb69d17be66cc4511a6b0078b8300d4ae1621c80dfc213f14e0"
Nov 02 02:15:19.200895 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:19.200862    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893095-86gp6"]
Nov 02 02:15:19.204508 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:19.204478    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893095-86gp6"]
Nov 02 02:15:19.216746 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:19.216711    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03aac5c9-0853-4f79-bc5d-f785b322ce0c" path="/var/lib/kubelet/pods/03aac5c9-0853-4f79-bc5d-f785b322ce0c/volumes"
Nov 02 02:15:21.291709 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.291681    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cmhhj"]
Nov 02 02:15:21.292167 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.292129    5639 cpu_manager.go:414] "containerMap: removing container" podUID="03aac5c9-0853-4f79-bc5d-f785b322ce0c" containerName="cleanup"
Nov 02 02:15:21.292167 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.292141    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03aac5c9-0853-4f79-bc5d-f785b322ce0c" containerName="cleanup"
Nov 02 02:15:21.292167 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.292159    5639 cpu_manager.go:414] "containerMap: removing container" podUID="03aac5c9-0853-4f79-bc5d-f785b322ce0c" containerName="cleanup"
Nov 02 02:15:21.292167 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.292165    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03aac5c9-0853-4f79-bc5d-f785b322ce0c" containerName="cleanup"
Nov 02 02:15:21.292370 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.292173    5639 cpu_manager.go:414] "containerMap: removing container" podUID="03aac5c9-0853-4f79-bc5d-f785b322ce0c" containerName="cleanup"
Nov 02 02:15:21.292370 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.292178    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03aac5c9-0853-4f79-bc5d-f785b322ce0c" containerName="cleanup"
Nov 02 02:15:21.292370 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.292257    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="03aac5c9-0853-4f79-bc5d-f785b322ce0c" containerName="cleanup"
Nov 02 02:15:21.292370 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.292268    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="03aac5c9-0853-4f79-bc5d-f785b322ce0c" containerName="cleanup"
Nov 02 02:15:21.292370 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.292274    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="03aac5c9-0853-4f79-bc5d-f785b322ce0c" containerName="cleanup"
Nov 02 02:15:21.296524 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.296509    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cmhhj"
Nov 02 02:15:21.353258 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.353218    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cmhhj"]
Nov 02 02:15:21.399666 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.399618    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:15:21.399973 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.399957    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:15:21.399973 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.399964    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:15:21.400155 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.400044    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:15:21.424706 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.424673    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17febcbd-d3c4-4b47-9877-948fb5607524-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-cmhhj\" (UID: \"17febcbd-d3c4-4b47-9877-948fb5607524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cmhhj"
Nov 02 02:15:21.424866 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.424735    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm7mr\" (UniqueName: \"kubernetes.io/projected/17febcbd-d3c4-4b47-9877-948fb5607524-kube-api-access-mm7mr\") pod \"maas-api-revoke-sub-85237e45-cmhhj\" (UID: \"17febcbd-d3c4-4b47-9877-948fb5607524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cmhhj"
Nov 02 02:15:21.424866 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.424843    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17febcbd-d3c4-4b47-9877-948fb5607524-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-cmhhj\" (UID: \"17febcbd-d3c4-4b47-9877-948fb5607524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cmhhj"
Nov 02 02:15:21.526501 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.526465    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm7mr\" (UniqueName: \"kubernetes.io/projected/17febcbd-d3c4-4b47-9877-948fb5607524-kube-api-access-mm7mr\") pod \"maas-api-revoke-sub-85237e45-cmhhj\" (UID: \"17febcbd-d3c4-4b47-9877-948fb5607524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cmhhj"
Nov 02 02:15:21.526707 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.526515    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17febcbd-d3c4-4b47-9877-948fb5607524-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-cmhhj\" (UID: \"17febcbd-d3c4-4b47-9877-948fb5607524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cmhhj"
Nov 02 02:15:21.526707 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.526569    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17febcbd-d3c4-4b47-9877-948fb5607524-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-cmhhj\" (UID: \"17febcbd-d3c4-4b47-9877-948fb5607524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cmhhj"
Nov 02 02:15:21.527311 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.527211    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17febcbd-d3c4-4b47-9877-948fb5607524-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-cmhhj\" (UID: \"17febcbd-d3c4-4b47-9877-948fb5607524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cmhhj"
Nov 02 02:15:21.528985 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.528964    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17febcbd-d3c4-4b47-9877-948fb5607524-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-cmhhj\" (UID: \"17febcbd-d3c4-4b47-9877-948fb5607524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cmhhj"
Nov 02 02:15:21.587557 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.587522    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm7mr\" (UniqueName: \"kubernetes.io/projected/17febcbd-d3c4-4b47-9877-948fb5607524-kube-api-access-mm7mr\") pod \"maas-api-revoke-sub-85237e45-cmhhj\" (UID: \"17febcbd-d3c4-4b47-9877-948fb5607524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cmhhj"
Nov 02 02:15:21.609379 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.609347    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cmhhj"
Nov 02 02:15:21.985071 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:21.985043    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cmhhj"]
Nov 02 02:15:21.988022 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:15:21.987986    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17febcbd_d3c4_4b47_9877_948fb5607524.slice/crio-0174eaead82eab5602cae7c7dbbff0ee6e61a2fbf8adf3a2ca74e6e40dd5a463 WatchSource:0}: Error finding container 0174eaead82eab5602cae7c7dbbff0ee6e61a2fbf8adf3a2ca74e6e40dd5a463: Status 404 returned error can't find the container with id 0174eaead82eab5602cae7c7dbbff0ee6e61a2fbf8adf3a2ca74e6e40dd5a463
Nov 02 02:15:22.056861 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:22.056818    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cmhhj" event={"ID":"17febcbd-d3c4-4b47-9877-948fb5607524","Type":"ContainerStarted","Data":"0174eaead82eab5602cae7c7dbbff0ee6e61a2fbf8adf3a2ca74e6e40dd5a463"}
Nov 02 02:15:23.065242 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:23.065202    5639 generic.go:356] "Generic (PLEG): container finished" podID="17febcbd-d3c4-4b47-9877-948fb5607524" containerID="ed73651922035a6c95887bf08d65659d023393185c424b9ada72c40e1c23d519" exitCode=0
Nov 02 02:15:23.065664 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:23.065300    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cmhhj" event={"ID":"17febcbd-d3c4-4b47-9877-948fb5607524","Type":"ContainerDied","Data":"ed73651922035a6c95887bf08d65659d023393185c424b9ada72c40e1c23d519"}
Nov 02 02:15:24.205074 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:24.205050    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cmhhj"
Nov 02 02:15:24.353896 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:24.353861    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17febcbd-d3c4-4b47-9877-948fb5607524-kube-api-access-mm7mr\" (UniqueName: \"kubernetes.io/projected/17febcbd-d3c4-4b47-9877-948fb5607524-kube-api-access-mm7mr\") pod \"17febcbd-d3c4-4b47-9877-948fb5607524\" (UID: \"17febcbd-d3c4-4b47-9877-948fb5607524\") "
Nov 02 02:15:24.354114 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:24.354007    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17febcbd-d3c4-4b47-9877-948fb5607524-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17febcbd-d3c4-4b47-9877-948fb5607524-maas-api-xks-ca\") pod \"17febcbd-d3c4-4b47-9877-948fb5607524\" (UID: \"17febcbd-d3c4-4b47-9877-948fb5607524\") "
Nov 02 02:15:24.354114 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:24.354055    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17febcbd-d3c4-4b47-9877-948fb5607524-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17febcbd-d3c4-4b47-9877-948fb5607524-maas-api-openshift-ca\") pod \"17febcbd-d3c4-4b47-9877-948fb5607524\" (UID: \"17febcbd-d3c4-4b47-9877-948fb5607524\") "
Nov 02 02:15:24.354548 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:24.354516    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17febcbd-d3c4-4b47-9877-948fb5607524-maas-api-openshift-ca" pod "17febcbd-d3c4-4b47-9877-948fb5607524" (UID: "17febcbd-d3c4-4b47-9877-948fb5607524"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:15:24.356929 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:24.356904    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17febcbd-d3c4-4b47-9877-948fb5607524-maas-api-xks-ca" pod "17febcbd-d3c4-4b47-9877-948fb5607524" (UID: "17febcbd-d3c4-4b47-9877-948fb5607524"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:15:24.357038 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:24.356938    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17febcbd-d3c4-4b47-9877-948fb5607524-kube-api-access-mm7mr" pod "17febcbd-d3c4-4b47-9877-948fb5607524" (UID: "17febcbd-d3c4-4b47-9877-948fb5607524"). InnerVolumeSpecName "kube-api-access-mm7mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:15:24.455895 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:24.455859    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17febcbd-d3c4-4b47-9877-948fb5607524-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:24.455895 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:24.455890    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm7mr\" (UniqueName: \"kubernetes.io/projected/17febcbd-d3c4-4b47-9877-948fb5607524-kube-api-access-mm7mr\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:24.456099 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:24.455905    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17febcbd-d3c4-4b47-9877-948fb5607524-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:25.078905 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:25.078876    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cmhhj"
Nov 02 02:15:25.079099 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:25.078871    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cmhhj" event={"ID":"17febcbd-d3c4-4b47-9877-948fb5607524","Type":"ContainerDied","Data":"0174eaead82eab5602cae7c7dbbff0ee6e61a2fbf8adf3a2ca74e6e40dd5a463"}
Nov 02 02:15:25.079099 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:25.079014    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0174eaead82eab5602cae7c7dbbff0ee6e61a2fbf8adf3a2ca74e6e40dd5a463"
Nov 02 02:15:27.423847 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:15:27.423808    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:15:33.858612 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:33.858571    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65b55dd8bb-482b8"]
Nov 02 02:15:33.859120 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:33.858815    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-65b55dd8bb-482b8" podUID="3cfc5266-30c3-4306-afbe-d1ae5c102df5" containerName="manager" containerID="cri-o://cc16f44a04ec59e1cd2007248be15da4382c72dc85e02ae50bab520ce6c9ae05" gracePeriod=30
Nov 02 02:15:34.101076 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:34.101051    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65b55dd8bb-482b8"
Nov 02 02:15:34.129697 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:34.129622    5639 generic.go:356] "Generic (PLEG): container finished" podID="3cfc5266-30c3-4306-afbe-d1ae5c102df5" containerID="cc16f44a04ec59e1cd2007248be15da4382c72dc85e02ae50bab520ce6c9ae05" exitCode=0
Nov 02 02:15:34.129697 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:34.129671    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65b55dd8bb-482b8"
Nov 02 02:15:34.129890 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:34.129708    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65b55dd8bb-482b8" event={"ID":"3cfc5266-30c3-4306-afbe-d1ae5c102df5","Type":"ContainerDied","Data":"cc16f44a04ec59e1cd2007248be15da4382c72dc85e02ae50bab520ce6c9ae05"}
Nov 02 02:15:34.129890 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:34.129763    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65b55dd8bb-482b8" event={"ID":"3cfc5266-30c3-4306-afbe-d1ae5c102df5","Type":"ContainerDied","Data":"7ef1cd530928566a911e9d3db02bf758dbd695711c2ab0d01fbb71bcea2c3141"}
Nov 02 02:15:34.129890 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:34.129793    5639 scope.go:122] "RemoveContainer" containerID="cc16f44a04ec59e1cd2007248be15da4382c72dc85e02ae50bab520ce6c9ae05"
Nov 02 02:15:34.139110 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:34.139093    5639 scope.go:122] "RemoveContainer" containerID="cc16f44a04ec59e1cd2007248be15da4382c72dc85e02ae50bab520ce6c9ae05"
Nov 02 02:15:34.139460 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:15:34.139434    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc16f44a04ec59e1cd2007248be15da4382c72dc85e02ae50bab520ce6c9ae05\": container with ID starting with cc16f44a04ec59e1cd2007248be15da4382c72dc85e02ae50bab520ce6c9ae05 not found: ID does not exist" containerID="cc16f44a04ec59e1cd2007248be15da4382c72dc85e02ae50bab520ce6c9ae05"
Nov 02 02:15:34.139541 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:34.139471    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc16f44a04ec59e1cd2007248be15da4382c72dc85e02ae50bab520ce6c9ae05"} err="failed to get container status \"cc16f44a04ec59e1cd2007248be15da4382c72dc85e02ae50bab520ce6c9ae05\": rpc error: code = NotFound desc = could not find container \"cc16f44a04ec59e1cd2007248be15da4382c72dc85e02ae50bab520ce6c9ae05\": container with ID starting with cc16f44a04ec59e1cd2007248be15da4382c72dc85e02ae50bab520ce6c9ae05 not found: ID does not exist"
Nov 02 02:15:34.264068 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:34.264035    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cfc5266-30c3-4306-afbe-d1ae5c102df5-webhook-cert\" (UniqueName: \"kubernetes.io/secret/3cfc5266-30c3-4306-afbe-d1ae5c102df5-webhook-cert\") pod \"3cfc5266-30c3-4306-afbe-d1ae5c102df5\" (UID: \"3cfc5266-30c3-4306-afbe-d1ae5c102df5\") "
Nov 02 02:15:34.264263 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:34.264140    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cfc5266-30c3-4306-afbe-d1ae5c102df5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cfc5266-30c3-4306-afbe-d1ae5c102df5-metrics-certs\") pod \"3cfc5266-30c3-4306-afbe-d1ae5c102df5\" (UID: \"3cfc5266-30c3-4306-afbe-d1ae5c102df5\") "
Nov 02 02:15:34.264263 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:34.264214    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cfc5266-30c3-4306-afbe-d1ae5c102df5-kube-api-access-kn92h\" (UniqueName: \"kubernetes.io/projected/3cfc5266-30c3-4306-afbe-d1ae5c102df5-kube-api-access-kn92h\") pod \"3cfc5266-30c3-4306-afbe-d1ae5c102df5\" (UID: \"3cfc5266-30c3-4306-afbe-d1ae5c102df5\") "
Nov 02 02:15:34.266497 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:34.266468    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cfc5266-30c3-4306-afbe-d1ae5c102df5-webhook-cert" pod "3cfc5266-30c3-4306-afbe-d1ae5c102df5" (UID: "3cfc5266-30c3-4306-afbe-d1ae5c102df5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:15:34.266617 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:34.266527    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cfc5266-30c3-4306-afbe-d1ae5c102df5-kube-api-access-kn92h" pod "3cfc5266-30c3-4306-afbe-d1ae5c102df5" (UID: "3cfc5266-30c3-4306-afbe-d1ae5c102df5"). InnerVolumeSpecName "kube-api-access-kn92h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:15:34.266617 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:34.266583    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cfc5266-30c3-4306-afbe-d1ae5c102df5-metrics-certs" pod "3cfc5266-30c3-4306-afbe-d1ae5c102df5" (UID: "3cfc5266-30c3-4306-afbe-d1ae5c102df5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:15:34.366067 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:34.366036    5639 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3cfc5266-30c3-4306-afbe-d1ae5c102df5-webhook-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:34.366067 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:34.366061    5639 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cfc5266-30c3-4306-afbe-d1ae5c102df5-metrics-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:34.366067 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:34.366070    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn92h\" (UniqueName: \"kubernetes.io/projected/3cfc5266-30c3-4306-afbe-d1ae5c102df5-kube-api-access-kn92h\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:34.554157 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:34.554121    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65b55dd8bb-482b8"]
Nov 02 02:15:34.558126 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:34.558103    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-65b55dd8bb-482b8"]
Nov 02 02:15:35.216862 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:35.216833    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cfc5266-30c3-4306-afbe-d1ae5c102df5" path="/var/lib/kubelet/pods/3cfc5266-30c3-4306-afbe-d1ae5c102df5/volumes"
Nov 02 02:15:35.967858 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:35.967828    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65b55dd8bb-mccfz"]
Nov 02 02:15:35.968229 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:35.968218    5639 cpu_manager.go:414] "containerMap: removing container" podUID="17febcbd-d3c4-4b47-9877-948fb5607524" containerName="revoke-keys"
Nov 02 02:15:35.968313 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:35.968231    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17febcbd-d3c4-4b47-9877-948fb5607524" containerName="revoke-keys"
Nov 02 02:15:35.968313 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:35.968253    5639 cpu_manager.go:414] "containerMap: removing container" podUID="3cfc5266-30c3-4306-afbe-d1ae5c102df5" containerName="manager"
Nov 02 02:15:35.968313 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:35.968258    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cfc5266-30c3-4306-afbe-d1ae5c102df5" containerName="manager"
Nov 02 02:15:35.968449 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:35.968359    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cfc5266-30c3-4306-afbe-d1ae5c102df5" containerName="manager"
Nov 02 02:15:35.968449 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:35.968380    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="17febcbd-d3c4-4b47-9877-948fb5607524" containerName="revoke-keys"
Nov 02 02:15:35.972765 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:35.972748    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65b55dd8bb-mccfz"
Nov 02 02:15:36.025355 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:36.025321    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 02:15:36.074667 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:36.074638    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ghnpp\""
Nov 02 02:15:36.074667 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:36.074638    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 02:15:36.074897 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:36.074678    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 02:15:36.076689 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:36.076665    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65b55dd8bb-mccfz"]
Nov 02 02:15:36.083697 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:36.083675    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b757d5cd-a65b-4f74-9ab5-4896809d8958-metrics-certs\") pod \"maas-controller-65b55dd8bb-mccfz\" (UID: \"b757d5cd-a65b-4f74-9ab5-4896809d8958\") " pod="opendatahub/maas-controller-65b55dd8bb-mccfz"
Nov 02 02:15:36.083831 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:36.083715    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b757d5cd-a65b-4f74-9ab5-4896809d8958-webhook-cert\") pod \"maas-controller-65b55dd8bb-mccfz\" (UID: \"b757d5cd-a65b-4f74-9ab5-4896809d8958\") " pod="opendatahub/maas-controller-65b55dd8bb-mccfz"
Nov 02 02:15:36.083831 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:36.083820    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlxtx\" (UniqueName: \"kubernetes.io/projected/b757d5cd-a65b-4f74-9ab5-4896809d8958-kube-api-access-mlxtx\") pod \"maas-controller-65b55dd8bb-mccfz\" (UID: \"b757d5cd-a65b-4f74-9ab5-4896809d8958\") " pod="opendatahub/maas-controller-65b55dd8bb-mccfz"
Nov 02 02:15:36.185247 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:36.185208    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b757d5cd-a65b-4f74-9ab5-4896809d8958-metrics-certs\") pod \"maas-controller-65b55dd8bb-mccfz\" (UID: \"b757d5cd-a65b-4f74-9ab5-4896809d8958\") " pod="opendatahub/maas-controller-65b55dd8bb-mccfz"
Nov 02 02:15:36.185439 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:36.185255    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b757d5cd-a65b-4f74-9ab5-4896809d8958-webhook-cert\") pod \"maas-controller-65b55dd8bb-mccfz\" (UID: \"b757d5cd-a65b-4f74-9ab5-4896809d8958\") " pod="opendatahub/maas-controller-65b55dd8bb-mccfz"
Nov 02 02:15:36.185439 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:36.185338    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlxtx\" (UniqueName: \"kubernetes.io/projected/b757d5cd-a65b-4f74-9ab5-4896809d8958-kube-api-access-mlxtx\") pod \"maas-controller-65b55dd8bb-mccfz\" (UID: \"b757d5cd-a65b-4f74-9ab5-4896809d8958\") " pod="opendatahub/maas-controller-65b55dd8bb-mccfz"
Nov 02 02:15:36.187842 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:36.187812    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b757d5cd-a65b-4f74-9ab5-4896809d8958-metrics-certs\") pod \"maas-controller-65b55dd8bb-mccfz\" (UID: \"b757d5cd-a65b-4f74-9ab5-4896809d8958\") " pod="opendatahub/maas-controller-65b55dd8bb-mccfz"
Nov 02 02:15:36.187966 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:36.187945    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b757d5cd-a65b-4f74-9ab5-4896809d8958-webhook-cert\") pod \"maas-controller-65b55dd8bb-mccfz\" (UID: \"b757d5cd-a65b-4f74-9ab5-4896809d8958\") " pod="opendatahub/maas-controller-65b55dd8bb-mccfz"
Nov 02 02:15:36.247081 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:36.247006    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlxtx\" (UniqueName: \"kubernetes.io/projected/b757d5cd-a65b-4f74-9ab5-4896809d8958-kube-api-access-mlxtx\") pod \"maas-controller-65b55dd8bb-mccfz\" (UID: \"b757d5cd-a65b-4f74-9ab5-4896809d8958\") " pod="opendatahub/maas-controller-65b55dd8bb-mccfz"
Nov 02 02:15:36.285713 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:36.285682    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65b55dd8bb-mccfz"
Nov 02 02:15:36.657317 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:36.657271    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65b55dd8bb-mccfz"]
Nov 02 02:15:36.659380 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:15:36.659351    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb757d5cd_a65b_4f74_9ab5_4896809d8958.slice/crio-d2de3536b8511591d42fc789fc6b28f26a0ef459990a9e446b3848e410478d62 WatchSource:0}: Error finding container d2de3536b8511591d42fc789fc6b28f26a0ef459990a9e446b3848e410478d62: Status 404 returned error can't find the container with id d2de3536b8511591d42fc789fc6b28f26a0ef459990a9e446b3848e410478d62
Nov 02 02:15:37.148170 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:37.148120    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65b55dd8bb-mccfz" event={"ID":"b757d5cd-a65b-4f74-9ab5-4896809d8958","Type":"ContainerStarted","Data":"d2de3536b8511591d42fc789fc6b28f26a0ef459990a9e446b3848e410478d62"}
Nov 02 02:15:38.155610 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:38.155570    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65b55dd8bb-mccfz" event={"ID":"b757d5cd-a65b-4f74-9ab5-4896809d8958","Type":"ContainerStarted","Data":"b14ab63c515bffd933c5ec4a4362e16b4fed41d4e958fb8a7deddb09cd5f6cc3"}
Nov 02 02:15:38.156037 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:38.155690    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65b55dd8bb-mccfz"
Nov 02 02:15:38.275654 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:38.275601    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-65b55dd8bb-mccfz" podStartSLOduration=2.437838126 podStartE2EDuration="3.275587947s" totalImagesPullingTime="837.749821ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:15:38.274447901 +0000 UTC m=+3003.178130038" watchObservedRunningTime="2026-11-02 02:15:38.275587947 +0000 UTC m=+3003.179270060"
Nov 02 02:15:39.138745 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:39.138711    5639 scope.go:122] "RemoveContainer" containerID="d0bbf64018dee4c5ce9f51e50bbc2ab168b86c5d42662568f7f5a8ff2302fcdc"
Nov 02 02:15:39.148314 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:39.148267    5639 scope.go:122] "RemoveContainer" containerID="77c18ec06259d9e36f6ad29bd63375b97eb790a22e0998d9a57c7c479fb4595b"
Nov 02 02:15:49.166524 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:15:49.166491    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65b55dd8bb-mccfz"
Nov 02 02:15:57.421611 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:15:57.421572    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:16:13.048675 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:13.048638    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c70ecb0c-g82hr"]
Nov 02 02:16:13.099930 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:13.099879    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdb6931-zhrmm"]
Nov 02 02:16:13.101375 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:13.101345    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c70ecb0c-g82hr"]
Nov 02 02:16:13.102711 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:13.102690    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdb6931-zhrmm"]
Nov 02 02:16:13.218078 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:13.218041    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78dba175-b3da-461b-b642-83a8fc049510" path="/var/lib/kubelet/pods/78dba175-b3da-461b-b642-83a8fc049510/volumes"
Nov 02 02:16:13.218512 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:13.218497    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e914afa8-6a65-486c-868e-f98369c26f6e" path="/var/lib/kubelet/pods/e914afa8-6a65-486c-868e-f98369c26f6e/volumes"
Nov 02 02:16:23.405072 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:23.405037    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:16:27.420801 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:16:27.420762    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:16:28.047723 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:28.047687    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38127a13-fkrm6"]
Nov 02 02:16:28.097682 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:28.097639    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee06273b-g8csb"]
Nov 02 02:16:28.099164 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:28.099135    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38127a13-fkrm6"]
Nov 02 02:16:28.100346 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:28.100322    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee06273b-g8csb"]
Nov 02 02:16:29.046321 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:29.046265    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01624761-nqfb5"]
Nov 02 02:16:29.096005 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:29.095971    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01624761-nqfb5"]
Nov 02 02:16:29.217904 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:29.217867    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="826a3aa6-79b4-4397-9f69-8563b45453e2" path="/var/lib/kubelet/pods/826a3aa6-79b4-4397-9f69-8563b45453e2/volumes"
Nov 02 02:16:29.218334 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:29.218319    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bccf3fd7-7bdd-4e45-9781-6c64d977c616" path="/var/lib/kubelet/pods/bccf3fd7-7bdd-4e45-9781-6c64d977c616/volumes"
Nov 02 02:16:29.218713 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:29.218700    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d97a1388-6ca5-4b25-8a9d-d651faf46527" path="/var/lib/kubelet/pods/d97a1388-6ca5-4b25-8a9d-d651faf46527/volumes"
Nov 02 02:16:31.969816 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:31.969788    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rvvf9"]
Nov 02 02:16:31.974670 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:31.974652    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rvvf9"
Nov 02 02:16:32.023248 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:32.023216    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhk4h\" (UniqueName: \"kubernetes.io/projected/99d4b035-486d-4d58-a5e3-2f4eafab38ca-kube-api-access-nhk4h\") pod \"maas-api-revoke-sub-c1e271a9-rvvf9\" (UID: \"99d4b035-486d-4d58-a5e3-2f4eafab38ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rvvf9"
Nov 02 02:16:32.023433 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:32.023355    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d4b035-486d-4d58-a5e3-2f4eafab38ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-rvvf9\" (UID: \"99d4b035-486d-4d58-a5e3-2f4eafab38ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rvvf9"
Nov 02 02:16:32.023433 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:32.023412    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d4b035-486d-4d58-a5e3-2f4eafab38ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-rvvf9\" (UID: \"99d4b035-486d-4d58-a5e3-2f4eafab38ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rvvf9"
Nov 02 02:16:32.078462 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:32.078428    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:16:32.078462 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:32.078445    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:16:32.078462 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:32.078454    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:16:32.078716 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:32.078515    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:16:32.080101 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:32.080079    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rvvf9"]
Nov 02 02:16:32.124860 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:32.124833    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d4b035-486d-4d58-a5e3-2f4eafab38ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-rvvf9\" (UID: \"99d4b035-486d-4d58-a5e3-2f4eafab38ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rvvf9"
Nov 02 02:16:32.125010 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:32.124879    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d4b035-486d-4d58-a5e3-2f4eafab38ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-rvvf9\" (UID: \"99d4b035-486d-4d58-a5e3-2f4eafab38ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rvvf9"
Nov 02 02:16:32.125010 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:32.124939    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhk4h\" (UniqueName: \"kubernetes.io/projected/99d4b035-486d-4d58-a5e3-2f4eafab38ca-kube-api-access-nhk4h\") pod \"maas-api-revoke-sub-c1e271a9-rvvf9\" (UID: \"99d4b035-486d-4d58-a5e3-2f4eafab38ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rvvf9"
Nov 02 02:16:32.125521 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:32.125500    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d4b035-486d-4d58-a5e3-2f4eafab38ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-rvvf9\" (UID: \"99d4b035-486d-4d58-a5e3-2f4eafab38ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rvvf9"
Nov 02 02:16:32.127393 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:32.127372    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d4b035-486d-4d58-a5e3-2f4eafab38ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-rvvf9\" (UID: \"99d4b035-486d-4d58-a5e3-2f4eafab38ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rvvf9"
Nov 02 02:16:32.185824 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:32.185788    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhk4h\" (UniqueName: \"kubernetes.io/projected/99d4b035-486d-4d58-a5e3-2f4eafab38ca-kube-api-access-nhk4h\") pod \"maas-api-revoke-sub-c1e271a9-rvvf9\" (UID: \"99d4b035-486d-4d58-a5e3-2f4eafab38ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rvvf9"
Nov 02 02:16:32.286965 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:32.286886    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rvvf9"
Nov 02 02:16:32.659479 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:32.659453    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rvvf9"]
Nov 02 02:16:32.662352 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:16:32.662308    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d4b035_486d_4d58_a5e3_2f4eafab38ca.slice/crio-c14c987727970e0693624a02b981f213c9f02e1beb36326939731ef9fb64a3ed WatchSource:0}: Error finding container c14c987727970e0693624a02b981f213c9f02e1beb36326939731ef9fb64a3ed: Status 404 returned error can't find the container with id c14c987727970e0693624a02b981f213c9f02e1beb36326939731ef9fb64a3ed
Nov 02 02:16:33.467007 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:33.466968    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rvvf9" event={"ID":"99d4b035-486d-4d58-a5e3-2f4eafab38ca","Type":"ContainerStarted","Data":"002b2b0a17d730268ce3554a74804716acc6428376e87b3a288e453d781dee5b"}
Nov 02 02:16:33.467432 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:33.467040    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rvvf9" event={"ID":"99d4b035-486d-4d58-a5e3-2f4eafab38ca","Type":"ContainerStarted","Data":"c14c987727970e0693624a02b981f213c9f02e1beb36326939731ef9fb64a3ed"}
Nov 02 02:16:33.583320 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:33.583252    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rvvf9" podStartSLOduration=2.583238595 podStartE2EDuration="2.583238595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:16:33.582098658 +0000 UTC m=+3058.485780777" watchObservedRunningTime="2026-11-02 02:16:33.583238595 +0000 UTC m=+3058.486920709"
Nov 02 02:16:34.473395 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:34.473353    5639 generic.go:356] "Generic (PLEG): container finished" podID="99d4b035-486d-4d58-a5e3-2f4eafab38ca" containerID="002b2b0a17d730268ce3554a74804716acc6428376e87b3a288e453d781dee5b" exitCode=0
Nov 02 02:16:34.473986 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:34.473446    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rvvf9" event={"ID":"99d4b035-486d-4d58-a5e3-2f4eafab38ca","Type":"ContainerDied","Data":"002b2b0a17d730268ce3554a74804716acc6428376e87b3a288e453d781dee5b"}
Nov 02 02:16:35.614969 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:35.614943    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rvvf9"
Nov 02 02:16:35.658748 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:35.658689    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99d4b035-486d-4d58-a5e3-2f4eafab38ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d4b035-486d-4d58-a5e3-2f4eafab38ca-maas-api-openshift-ca\") pod \"99d4b035-486d-4d58-a5e3-2f4eafab38ca\" (UID: \"99d4b035-486d-4d58-a5e3-2f4eafab38ca\") "
Nov 02 02:16:35.658978 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:35.658765    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99d4b035-486d-4d58-a5e3-2f4eafab38ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d4b035-486d-4d58-a5e3-2f4eafab38ca-maas-api-xks-ca\") pod \"99d4b035-486d-4d58-a5e3-2f4eafab38ca\" (UID: \"99d4b035-486d-4d58-a5e3-2f4eafab38ca\") "
Nov 02 02:16:35.658978 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:35.658894    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99d4b035-486d-4d58-a5e3-2f4eafab38ca-kube-api-access-nhk4h\" (UniqueName: \"kubernetes.io/projected/99d4b035-486d-4d58-a5e3-2f4eafab38ca-kube-api-access-nhk4h\") pod \"99d4b035-486d-4d58-a5e3-2f4eafab38ca\" (UID: \"99d4b035-486d-4d58-a5e3-2f4eafab38ca\") "
Nov 02 02:16:35.659190 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:35.659159    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d4b035-486d-4d58-a5e3-2f4eafab38ca-maas-api-openshift-ca" pod "99d4b035-486d-4d58-a5e3-2f4eafab38ca" (UID: "99d4b035-486d-4d58-a5e3-2f4eafab38ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:16:35.661074 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:35.661046    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99d4b035-486d-4d58-a5e3-2f4eafab38ca-maas-api-xks-ca" pod "99d4b035-486d-4d58-a5e3-2f4eafab38ca" (UID: "99d4b035-486d-4d58-a5e3-2f4eafab38ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:16:35.661232 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:35.661098    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d4b035-486d-4d58-a5e3-2f4eafab38ca-kube-api-access-nhk4h" pod "99d4b035-486d-4d58-a5e3-2f4eafab38ca" (UID: "99d4b035-486d-4d58-a5e3-2f4eafab38ca"). InnerVolumeSpecName "kube-api-access-nhk4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:16:35.760296 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:35.760177    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhk4h\" (UniqueName: \"kubernetes.io/projected/99d4b035-486d-4d58-a5e3-2f4eafab38ca-kube-api-access-nhk4h\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:16:35.760296 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:35.760209    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d4b035-486d-4d58-a5e3-2f4eafab38ca-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:16:35.760296 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:35.760219    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d4b035-486d-4d58-a5e3-2f4eafab38ca-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:16:36.485223 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:36.485191    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rvvf9"
Nov 02 02:16:36.485433 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:36.485234    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rvvf9" event={"ID":"99d4b035-486d-4d58-a5e3-2f4eafab38ca","Type":"ContainerDied","Data":"c14c987727970e0693624a02b981f213c9f02e1beb36326939731ef9fb64a3ed"}
Nov 02 02:16:36.485433 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:36.485314    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c14c987727970e0693624a02b981f213c9f02e1beb36326939731ef9fb64a3ed"
Nov 02 02:16:39.193847 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:39.193817    5639 scope.go:122] "RemoveContainer" containerID="f01cf1fb575f62d8f81b4be52e253eb797b54e9fd6a2dd74408accb0422191b1"
Nov 02 02:16:39.827724 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:39.827695    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"]
Nov 02 02:16:39.828142 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:39.828129    5639 cpu_manager.go:414] "containerMap: removing container" podUID="99d4b035-486d-4d58-a5e3-2f4eafab38ca" containerName="revoke-keys"
Nov 02 02:16:39.828189 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:39.828144    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99d4b035-486d-4d58-a5e3-2f4eafab38ca" containerName="revoke-keys"
Nov 02 02:16:39.828240 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:39.828230    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="99d4b035-486d-4d58-a5e3-2f4eafab38ca" containerName="revoke-keys"
Nov 02 02:16:39.832437 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:39.832417    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:39.877364 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:39.877328    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"]
Nov 02 02:16:39.881049 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:39.881026    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:39.899734 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:39.899697    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e6c6d59-73b1-4b69-94d5-586b38902af4-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:39.899942 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:39.899753    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e6c6d59-73b1-4b69-94d5-586b38902af4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:39.899942 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:39.899778    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e6c6d59-73b1-4b69-94d5-586b38902af4-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:39.899942 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:39.899899    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e6c6d59-73b1-4b69-94d5-586b38902af4-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:39.899942 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:39.899932    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e6c6d59-73b1-4b69-94d5-586b38902af4-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:39.900157 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:39.899996    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whp7c\" (UniqueName: \"kubernetes.io/projected/9e6c6d59-73b1-4b69-94d5-586b38902af4-kube-api-access-whp7c\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:39.900157 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:39.900056    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e6c6d59-73b1-4b69-94d5-586b38902af4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:39.900157 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:39.900106    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e6c6d59-73b1-4b69-94d5-586b38902af4-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:39.900292 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:39.900170    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e6c6d59-73b1-4b69-94d5-586b38902af4-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:39.977886 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:39.977850    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"]
Nov 02 02:16:40.001651 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.001612    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faf2916d-803b-4f14-a2b1-1c45b742fc4d-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.001869 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.001662    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e6c6d59-73b1-4b69-94d5-586b38902af4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:40.001869 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.001749    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e6c6d59-73b1-4b69-94d5-586b38902af4-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:40.001869 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.001785    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e6c6d59-73b1-4b69-94d5-586b38902af4-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:40.002060 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.001920    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e6c6d59-73b1-4b69-94d5-586b38902af4-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:40.002060 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.001987    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faf2916d-803b-4f14-a2b1-1c45b742fc4d-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.002060 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.002025    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf2916d-803b-4f14-a2b1-1c45b742fc4d-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.002234 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.002059    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e6c6d59-73b1-4b69-94d5-586b38902af4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:40.002325 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.002229    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faf2916d-803b-4f14-a2b1-1c45b742fc4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.002325 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.002252    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e6c6d59-73b1-4b69-94d5-586b38902af4-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:40.002325 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.002265    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e6c6d59-73b1-4b69-94d5-586b38902af4-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:40.002679 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.002336    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faf2916d-803b-4f14-a2b1-1c45b742fc4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.002679 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.002337    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e6c6d59-73b1-4b69-94d5-586b38902af4-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:40.002679 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.002448    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e6c6d59-73b1-4b69-94d5-586b38902af4-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:40.002679 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.002495    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccx6k\" (UniqueName: \"kubernetes.io/projected/faf2916d-803b-4f14-a2b1-1c45b742fc4d-kube-api-access-ccx6k\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.002679 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.002523    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e6c6d59-73b1-4b69-94d5-586b38902af4-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:40.002679 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.002585    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf2916d-803b-4f14-a2b1-1c45b742fc4d-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.002679 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.002589    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e6c6d59-73b1-4b69-94d5-586b38902af4-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:40.002679 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.002645    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whp7c\" (UniqueName: \"kubernetes.io/projected/9e6c6d59-73b1-4b69-94d5-586b38902af4-kube-api-access-whp7c\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:40.002679 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.002673    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faf2916d-803b-4f14-a2b1-1c45b742fc4d-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.003201 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.002698    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faf2916d-803b-4f14-a2b1-1c45b742fc4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.003201 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.002748    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e6c6d59-73b1-4b69-94d5-586b38902af4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:40.003201 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.003015    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e6c6d59-73b1-4b69-94d5-586b38902af4-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:40.004462 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.004440    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e6c6d59-73b1-4b69-94d5-586b38902af4-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:40.004590 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.004574    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e6c6d59-73b1-4b69-94d5-586b38902af4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:40.033899 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.033862    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"]
Nov 02 02:16:40.065730 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.065692    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e6c6d59-73b1-4b69-94d5-586b38902af4-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:40.066517 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.066478    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whp7c\" (UniqueName: \"kubernetes.io/projected/9e6c6d59-73b1-4b69-94d5-586b38902af4-kube-api-access-whp7c\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whtg2\" (UID: \"9e6c6d59-73b1-4b69-94d5-586b38902af4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:40.104190 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.104101    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf2916d-803b-4f14-a2b1-1c45b742fc4d-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.104190 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.104148    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faf2916d-803b-4f14-a2b1-1c45b742fc4d-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.104190 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.104166    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faf2916d-803b-4f14-a2b1-1c45b742fc4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.104190 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.104186    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faf2916d-803b-4f14-a2b1-1c45b742fc4d-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.104593 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.104260    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faf2916d-803b-4f14-a2b1-1c45b742fc4d-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.104593 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.104323    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf2916d-803b-4f14-a2b1-1c45b742fc4d-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.104593 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.104366    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faf2916d-803b-4f14-a2b1-1c45b742fc4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.104593 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.104390    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faf2916d-803b-4f14-a2b1-1c45b742fc4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.104593 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.104422    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccx6k\" (UniqueName: \"kubernetes.io/projected/faf2916d-803b-4f14-a2b1-1c45b742fc4d-kube-api-access-ccx6k\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.104593 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.104559    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faf2916d-803b-4f14-a2b1-1c45b742fc4d-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.104852 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.104609    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf2916d-803b-4f14-a2b1-1c45b742fc4d-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.105018 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.104998    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf2916d-803b-4f14-a2b1-1c45b742fc4d-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.105106 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.105033    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faf2916d-803b-4f14-a2b1-1c45b742fc4d-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.105794 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.105769    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faf2916d-803b-4f14-a2b1-1c45b742fc4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.106954 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.106929    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faf2916d-803b-4f14-a2b1-1c45b742fc4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.107081 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.107064    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faf2916d-803b-4f14-a2b1-1c45b742fc4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.145432 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.145387    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:40.167479 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.167450    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faf2916d-803b-4f14-a2b1-1c45b742fc4d-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.167705 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.167678    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccx6k\" (UniqueName: \"kubernetes.io/projected/faf2916d-803b-4f14-a2b1-1c45b742fc4d-kube-api-access-ccx6k\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fd65x\" (UID: \"faf2916d-803b-4f14-a2b1-1c45b742fc4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.194413 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.194365    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:40.572324 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.572274    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"]
Nov 02 02:16:40.574248 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:16:40.574221    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6c6d59_73b1_4b69_94d5_586b38902af4.slice/crio-4a261dbf12f98707309183dbc6135f36bb9b26b0fcda7e89e4373cb389a3a06d WatchSource:0}: Error finding container 4a261dbf12f98707309183dbc6135f36bb9b26b0fcda7e89e4373cb389a3a06d: Status 404 returned error can't find the container with id 4a261dbf12f98707309183dbc6135f36bb9b26b0fcda7e89e4373cb389a3a06d
Nov 02 02:16:40.576494 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.576462    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:16:40.576568 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.576530    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:16:40.576568 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.576557    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:16:40.625142 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.625113    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"]
Nov 02 02:16:40.628242 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:16:40.628210    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaf2916d_803b_4f14_a2b1_1c45b742fc4d.slice/crio-e7ef51eaf62da688376900e5eeeddac5eb74f73ec2b794140664feed243b1b85 WatchSource:0}: Error finding container e7ef51eaf62da688376900e5eeeddac5eb74f73ec2b794140664feed243b1b85: Status 404 returned error can't find the container with id e7ef51eaf62da688376900e5eeeddac5eb74f73ec2b794140664feed243b1b85
Nov 02 02:16:40.630654 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.630621    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:16:40.630752 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.630709    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:16:40.630826 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:40.630753    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:16:41.516218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:41.516175    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2" event={"ID":"9e6c6d59-73b1-4b69-94d5-586b38902af4","Type":"ContainerStarted","Data":"97f5fc3ba2a64123682d8c89bf08216739c7f29a5b8b29a298c8722c3e31f33f"}
Nov 02 02:16:41.516642 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:41.516247    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2" event={"ID":"9e6c6d59-73b1-4b69-94d5-586b38902af4","Type":"ContainerStarted","Data":"4a261dbf12f98707309183dbc6135f36bb9b26b0fcda7e89e4373cb389a3a06d"}
Nov 02 02:16:41.518047 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:41.518022    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x" event={"ID":"faf2916d-803b-4f14-a2b1-1c45b742fc4d","Type":"ContainerStarted","Data":"2f283307446254a39c242c7b2f0793cb3123f94942bdc0a6381acca2519ddb1c"}
Nov 02 02:16:41.518104 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:41.518068    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x" event={"ID":"faf2916d-803b-4f14-a2b1-1c45b742fc4d","Type":"ContainerStarted","Data":"e7ef51eaf62da688376900e5eeeddac5eb74f73ec2b794140664feed243b1b85"}
Nov 02 02:16:41.642289 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:41.642223    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2" podStartSLOduration=2.6422076 podStartE2EDuration="2.6422076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:16:41.640514718 +0000 UTC m=+3066.544196832" watchObservedRunningTime="2026-11-02 02:16:41.6422076 +0000 UTC m=+3066.545889712"
Nov 02 02:16:41.783366 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:41.782270    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x" podStartSLOduration=2.782248703 podStartE2EDuration="2.782248703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:16:41.779296219 +0000 UTC m=+3066.682978332" watchObservedRunningTime="2026-11-02 02:16:41.782248703 +0000 UTC m=+3066.685930818"
Nov 02 02:16:42.146259 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:42.146221    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:42.147720 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:42.147694    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-whtg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" start-of-body=
Nov 02 02:16:42.147857 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:42.147739    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2" podUID="9e6c6d59-73b1-4b69-94d5-586b38902af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused"
Nov 02 02:16:42.197659 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:42.195712    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:42.198002 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:42.197978    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fd65x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Nov 02 02:16:42.198187 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:42.198161    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x" podUID="faf2916d-803b-4f14-a2b1-1c45b742fc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Nov 02 02:16:43.145993 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:43.145954    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-whtg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" start-of-body=
Nov 02 02:16:43.146469 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:43.146013    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2" podUID="9e6c6d59-73b1-4b69-94d5-586b38902af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused"
Nov 02 02:16:43.196784 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:43.196740    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fd65x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Nov 02 02:16:43.197126 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:43.197093    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x" podUID="faf2916d-803b-4f14-a2b1-1c45b742fc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Nov 02 02:16:44.146718 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:44.146682    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-whtg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" start-of-body=
Nov 02 02:16:44.147166 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:44.146737    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2" podUID="9e6c6d59-73b1-4b69-94d5-586b38902af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused"
Nov 02 02:16:44.196308 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:44.195704    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fd65x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Nov 02 02:16:44.196308 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:44.195769    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x" podUID="faf2916d-803b-4f14-a2b1-1c45b742fc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Nov 02 02:16:45.146785 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:45.146745    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-whtg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" start-of-body=
Nov 02 02:16:45.147200 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:45.146808    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2" podUID="9e6c6d59-73b1-4b69-94d5-586b38902af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused"
Nov 02 02:16:45.198728 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:45.198688    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fd65x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Nov 02 02:16:45.199067 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:45.199011    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x" podUID="faf2916d-803b-4f14-a2b1-1c45b742fc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Nov 02 02:16:45.239170 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:45.239127    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:16:46.146144 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:46.146112    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-whtg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" start-of-body=
Nov 02 02:16:46.146370 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:46.146167    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2" podUID="9e6c6d59-73b1-4b69-94d5-586b38902af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused"
Nov 02 02:16:46.197305 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:46.195913    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fd65x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Nov 02 02:16:46.197305 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:46.195976    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x" podUID="faf2916d-803b-4f14-a2b1-1c45b742fc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Nov 02 02:16:47.146350 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:47.146271    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-whtg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" start-of-body=
Nov 02 02:16:47.146527 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:47.146388    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2" podUID="9e6c6d59-73b1-4b69-94d5-586b38902af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused"
Nov 02 02:16:47.195665 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:47.195622    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fd65x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Nov 02 02:16:47.195835 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:47.195691    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x" podUID="faf2916d-803b-4f14-a2b1-1c45b742fc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Nov 02 02:16:48.146298 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:48.146253    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-whtg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" start-of-body=
Nov 02 02:16:48.146840 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:48.146318    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2" podUID="9e6c6d59-73b1-4b69-94d5-586b38902af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused"
Nov 02 02:16:48.195366 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:48.195322    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fd65x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Nov 02 02:16:48.195561 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:48.195398    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x" podUID="faf2916d-803b-4f14-a2b1-1c45b742fc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Nov 02 02:16:50.187510 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:50.187450    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-whtg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:16:50.188058 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:50.187553    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2" podUID="9e6c6d59-73b1-4b69-94d5-586b38902af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:16:50.236454 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:50.236412    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fd65x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:16:50.236632 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:50.236491    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x" podUID="faf2916d-803b-4f14-a2b1-1c45b742fc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:16:51.188651 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:51.188606    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-whtg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:16:51.189201 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:51.188690    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2" podUID="9e6c6d59-73b1-4b69-94d5-586b38902af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:16:51.277530 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:51.277485    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fd65x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:16:51.277754 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:51.277562    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x" podUID="faf2916d-803b-4f14-a2b1-1c45b742fc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:16:52.191308 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:52.190384    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-whtg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:16:52.191308 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:52.190484    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2" podUID="9e6c6d59-73b1-4b69-94d5-586b38902af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:16:52.279427 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:52.279383    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fd65x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:16:52.279767 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:52.279736    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x" podUID="faf2916d-803b-4f14-a2b1-1c45b742fc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:16:53.052551 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:53.052517    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d77bda71-vqg7l"]
Nov 02 02:16:53.110306 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:53.109790    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d77bda71-vqg7l"]
Nov 02 02:16:53.242301 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:53.236383    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-whtg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:16:53.242301 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:53.236462    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2" podUID="9e6c6d59-73b1-4b69-94d5-586b38902af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:16:53.322544 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:53.322438    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fd65x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:16:53.326432 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:53.326356    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x" podUID="faf2916d-803b-4f14-a2b1-1c45b742fc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:16:53.342331 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:53.340635    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdb9bf00-7cc7-4476-a8f5-53aab6d658e0" path="/var/lib/kubelet/pods/fdb9bf00-7cc7-4476-a8f5-53aab6d658e0/volumes"
Nov 02 02:16:53.872108 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:53.872071    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:53.872577 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:53.872557    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:53.986464 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:53.986435    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2"
Nov 02 02:16:54.321637 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:54.321556    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:54.322060 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:54.321835    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:54.323118 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:54.323101    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x"
Nov 02 02:16:56.408788 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:16:56.408749    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:16:57.421465 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:16:57.421421    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:17:03.260806 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:03.260774    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8v2f6"]
Nov 02 02:17:03.265717 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:03.265697    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8v2f6"
Nov 02 02:17:03.321831 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:03.321786    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8v2f6"]
Nov 02 02:17:03.346546 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:03.346512    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8769577d-5dcc-4bd3-8735-3e25e2db4a25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-8v2f6\" (UID: \"8769577d-5dcc-4bd3-8735-3e25e2db4a25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8v2f6"
Nov 02 02:17:03.346744 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:03.346559    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8769577d-5dcc-4bd3-8735-3e25e2db4a25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-8v2f6\" (UID: \"8769577d-5dcc-4bd3-8735-3e25e2db4a25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8v2f6"
Nov 02 02:17:03.346744 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:03.346626    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j77cz\" (UniqueName: \"kubernetes.io/projected/8769577d-5dcc-4bd3-8735-3e25e2db4a25-kube-api-access-j77cz\") pod \"maas-api-revoke-sub-2e1f5b5f-8v2f6\" (UID: \"8769577d-5dcc-4bd3-8735-3e25e2db4a25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8v2f6"
Nov 02 02:17:03.369599 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:03.369569    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:17:03.369734 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:03.369606    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:17:03.369786 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:03.369767    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:17:03.369826 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:03.369774    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:17:03.447700 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:03.447669    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8769577d-5dcc-4bd3-8735-3e25e2db4a25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-8v2f6\" (UID: \"8769577d-5dcc-4bd3-8735-3e25e2db4a25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8v2f6"
Nov 02 02:17:03.447863 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:03.447708    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8769577d-5dcc-4bd3-8735-3e25e2db4a25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-8v2f6\" (UID: \"8769577d-5dcc-4bd3-8735-3e25e2db4a25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8v2f6"
Nov 02 02:17:03.447863 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:03.447750    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j77cz\" (UniqueName: \"kubernetes.io/projected/8769577d-5dcc-4bd3-8735-3e25e2db4a25-kube-api-access-j77cz\") pod \"maas-api-revoke-sub-2e1f5b5f-8v2f6\" (UID: \"8769577d-5dcc-4bd3-8735-3e25e2db4a25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8v2f6"
Nov 02 02:17:03.448454 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:03.448429    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8769577d-5dcc-4bd3-8735-3e25e2db4a25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-8v2f6\" (UID: \"8769577d-5dcc-4bd3-8735-3e25e2db4a25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8v2f6"
Nov 02 02:17:03.450324 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:03.450295    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8769577d-5dcc-4bd3-8735-3e25e2db4a25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-8v2f6\" (UID: \"8769577d-5dcc-4bd3-8735-3e25e2db4a25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8v2f6"
Nov 02 02:17:03.508823 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:03.508789    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j77cz\" (UniqueName: \"kubernetes.io/projected/8769577d-5dcc-4bd3-8735-3e25e2db4a25-kube-api-access-j77cz\") pod \"maas-api-revoke-sub-2e1f5b5f-8v2f6\" (UID: \"8769577d-5dcc-4bd3-8735-3e25e2db4a25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8v2f6"
Nov 02 02:17:03.579466 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:03.579441    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8v2f6"
Nov 02 02:17:03.957620 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:03.957592    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8v2f6"]
Nov 02 02:17:04.683479 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:04.683442    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8v2f6" event={"ID":"8769577d-5dcc-4bd3-8735-3e25e2db4a25","Type":"ContainerStarted","Data":"2e9562ea578d40b1771fed6e4fe43ef5840bd227e5d66acd5134eedcf53063a0"}
Nov 02 02:17:04.683948 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:04.683503    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8v2f6" event={"ID":"8769577d-5dcc-4bd3-8735-3e25e2db4a25","Type":"ContainerStarted","Data":"cb2ef04551889d166804d7cecd3fc7250c9eeff3a57ee72ea7b5f3931f4ba533"}
Nov 02 02:17:04.800201 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:04.800139    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8v2f6" podStartSLOduration=1.800123361 podStartE2EDuration="1.800123361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:17:04.798302384 +0000 UTC m=+3089.701984501" watchObservedRunningTime="2026-11-02 02:17:04.800123361 +0000 UTC m=+3089.703805475"
Nov 02 02:17:05.691107 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:05.691069    5639 generic.go:356] "Generic (PLEG): container finished" podID="8769577d-5dcc-4bd3-8735-3e25e2db4a25" containerID="2e9562ea578d40b1771fed6e4fe43ef5840bd227e5d66acd5134eedcf53063a0" exitCode=0
Nov 02 02:17:05.691598 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:05.691168    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8v2f6" event={"ID":"8769577d-5dcc-4bd3-8735-3e25e2db4a25","Type":"ContainerDied","Data":"2e9562ea578d40b1771fed6e4fe43ef5840bd227e5d66acd5134eedcf53063a0"}
Nov 02 02:17:06.835455 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:06.835419    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8v2f6"
Nov 02 02:17:06.880484 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:06.880449    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8769577d-5dcc-4bd3-8735-3e25e2db4a25-kube-api-access-j77cz\" (UniqueName: \"kubernetes.io/projected/8769577d-5dcc-4bd3-8735-3e25e2db4a25-kube-api-access-j77cz\") pod \"8769577d-5dcc-4bd3-8735-3e25e2db4a25\" (UID: \"8769577d-5dcc-4bd3-8735-3e25e2db4a25\") "
Nov 02 02:17:06.880669 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:06.880496    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8769577d-5dcc-4bd3-8735-3e25e2db4a25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8769577d-5dcc-4bd3-8735-3e25e2db4a25-maas-api-xks-ca\") pod \"8769577d-5dcc-4bd3-8735-3e25e2db4a25\" (UID: \"8769577d-5dcc-4bd3-8735-3e25e2db4a25\") "
Nov 02 02:17:06.880669 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:06.880593    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8769577d-5dcc-4bd3-8735-3e25e2db4a25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8769577d-5dcc-4bd3-8735-3e25e2db4a25-maas-api-openshift-ca\") pod \"8769577d-5dcc-4bd3-8735-3e25e2db4a25\" (UID: \"8769577d-5dcc-4bd3-8735-3e25e2db4a25\") "
Nov 02 02:17:06.880970 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:06.880941    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8769577d-5dcc-4bd3-8735-3e25e2db4a25-maas-api-openshift-ca" pod "8769577d-5dcc-4bd3-8735-3e25e2db4a25" (UID: "8769577d-5dcc-4bd3-8735-3e25e2db4a25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:17:06.882756 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:06.882729    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8769577d-5dcc-4bd3-8735-3e25e2db4a25-kube-api-access-j77cz" pod "8769577d-5dcc-4bd3-8735-3e25e2db4a25" (UID: "8769577d-5dcc-4bd3-8735-3e25e2db4a25"). InnerVolumeSpecName "kube-api-access-j77cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:17:06.882756 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:06.882744    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8769577d-5dcc-4bd3-8735-3e25e2db4a25-maas-api-xks-ca" pod "8769577d-5dcc-4bd3-8735-3e25e2db4a25" (UID: "8769577d-5dcc-4bd3-8735-3e25e2db4a25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:17:06.981853 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:06.981764    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j77cz\" (UniqueName: \"kubernetes.io/projected/8769577d-5dcc-4bd3-8735-3e25e2db4a25-kube-api-access-j77cz\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:06.981853 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:06.981793    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8769577d-5dcc-4bd3-8735-3e25e2db4a25-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:06.981853 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:06.981802    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8769577d-5dcc-4bd3-8735-3e25e2db4a25-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:07.705055 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:07.705024    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8v2f6"
Nov 02 02:17:07.705055 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:07.705034    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8v2f6" event={"ID":"8769577d-5dcc-4bd3-8735-3e25e2db4a25","Type":"ContainerDied","Data":"cb2ef04551889d166804d7cecd3fc7250c9eeff3a57ee72ea7b5f3931f4ba533"}
Nov 02 02:17:07.705375 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:07.705082    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb2ef04551889d166804d7cecd3fc7250c9eeff3a57ee72ea7b5f3931f4ba533"
Nov 02 02:17:15.641695 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:15.641661    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4sgcz"]
Nov 02 02:17:15.642166 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:15.642092    5639 cpu_manager.go:414] "containerMap: removing container" podUID="8769577d-5dcc-4bd3-8735-3e25e2db4a25" containerName="revoke-keys"
Nov 02 02:17:15.642166 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:15.642104    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8769577d-5dcc-4bd3-8735-3e25e2db4a25" containerName="revoke-keys"
Nov 02 02:17:15.642250 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:15.642192    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="8769577d-5dcc-4bd3-8735-3e25e2db4a25" containerName="revoke-keys"
Nov 02 02:17:15.645407 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:15.645388    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4sgcz"
Nov 02 02:17:15.697830 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:15.697797    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:17:15.698003 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:15.697906    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:17:15.747478 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:15.747449    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:17:15.747478 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:15.747480    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:17:15.749354 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:15.749334    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4sgcz"]
Nov 02 02:17:15.767205 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:15.767180    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4sgcz\" (UID: \"e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4sgcz"
Nov 02 02:17:15.767366 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:15.767223    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4sgcz\" (UID: \"e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4sgcz"
Nov 02 02:17:15.767366 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:15.767316    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ggcp\" (UniqueName: \"kubernetes.io/projected/e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4-kube-api-access-9ggcp\") pod \"maas-api-revoke-sub-ec843018-4sgcz\" (UID: \"e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4sgcz"
Nov 02 02:17:15.868738 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:15.868703    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4sgcz\" (UID: \"e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4sgcz"
Nov 02 02:17:15.868738 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:15.868747    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ggcp\" (UniqueName: \"kubernetes.io/projected/e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4-kube-api-access-9ggcp\") pod \"maas-api-revoke-sub-ec843018-4sgcz\" (UID: \"e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4sgcz"
Nov 02 02:17:15.869036 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:15.868835    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4sgcz\" (UID: \"e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4sgcz"
Nov 02 02:17:15.869503 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:15.869477    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4sgcz\" (UID: \"e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4sgcz"
Nov 02 02:17:15.871343 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:15.871322    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4sgcz\" (UID: \"e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4sgcz"
Nov 02 02:17:15.930656 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:15.930587    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ggcp\" (UniqueName: \"kubernetes.io/projected/e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4-kube-api-access-9ggcp\") pod \"maas-api-revoke-sub-ec843018-4sgcz\" (UID: \"e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4sgcz"
Nov 02 02:17:15.962801 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:15.962768    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4sgcz"
Nov 02 02:17:16.336105 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:16.336071    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4sgcz"]
Nov 02 02:17:16.338500 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:17:16.338465    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b9fec5_2f5c_4d28_adb8_65e2f4f0e7f4.slice/crio-cf1edfe4c410139191d3d6383451ecf89ab4da969d9edeb95b4274e34495ad26 WatchSource:0}: Error finding container cf1edfe4c410139191d3d6383451ecf89ab4da969d9edeb95b4274e34495ad26: Status 404 returned error can't find the container with id cf1edfe4c410139191d3d6383451ecf89ab4da969d9edeb95b4274e34495ad26
Nov 02 02:17:16.755466 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:16.755427    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4sgcz" event={"ID":"e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4","Type":"ContainerStarted","Data":"e30be2785045befb8df3d2c5e08450da9bb0505fd2c5d0f5dac737ffbe196628"}
Nov 02 02:17:16.755905 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:16.755486    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4sgcz" event={"ID":"e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4","Type":"ContainerStarted","Data":"cf1edfe4c410139191d3d6383451ecf89ab4da969d9edeb95b4274e34495ad26"}
Nov 02 02:17:16.873254 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:16.873186    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4sgcz" podStartSLOduration=1.873166589 podStartE2EDuration="1.873166589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:17:16.871154743 +0000 UTC m=+3101.774836860" watchObservedRunningTime="2026-11-02 02:17:16.873166589 +0000 UTC m=+3101.776848703"
Nov 02 02:17:17.762818 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:17.762782    5639 generic.go:356] "Generic (PLEG): container finished" podID="e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4" containerID="e30be2785045befb8df3d2c5e08450da9bb0505fd2c5d0f5dac737ffbe196628" exitCode=0
Nov 02 02:17:17.763241 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:17.762874    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4sgcz" event={"ID":"e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4","Type":"ContainerDied","Data":"e30be2785045befb8df3d2c5e08450da9bb0505fd2c5d0f5dac737ffbe196628"}
Nov 02 02:17:18.902809 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:18.902783    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4sgcz"
Nov 02 02:17:19.000470 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:19.000429    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4-maas-api-xks-ca\") pod \"e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4\" (UID: \"e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4\") "
Nov 02 02:17:19.000660 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:19.000532    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4-maas-api-openshift-ca\") pod \"e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4\" (UID: \"e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4\") "
Nov 02 02:17:19.000660 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:19.000559    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4-kube-api-access-9ggcp\" (UniqueName: \"kubernetes.io/projected/e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4-kube-api-access-9ggcp\") pod \"e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4\" (UID: \"e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4\") "
Nov 02 02:17:19.000989 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:19.000963    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4-maas-api-openshift-ca" pod "e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4" (UID: "e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:17:19.002752 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:19.002724    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4-maas-api-xks-ca" pod "e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4" (UID: "e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:17:19.002857 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:19.002837    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4-kube-api-access-9ggcp" pod "e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4" (UID: "e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4"). InnerVolumeSpecName "kube-api-access-9ggcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:17:19.101967 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:19.101933    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:19.101967 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:19.101962    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:19.102187 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:19.101975    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ggcp\" (UniqueName: \"kubernetes.io/projected/e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4-kube-api-access-9ggcp\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:19.669592 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:19.669543    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:17:19.775815 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:19.775779    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4sgcz"
Nov 02 02:17:19.775815 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:19.775805    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4sgcz" event={"ID":"e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4","Type":"ContainerDied","Data":"cf1edfe4c410139191d3d6383451ecf89ab4da969d9edeb95b4274e34495ad26"}
Nov 02 02:17:19.776036 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:19.775857    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf1edfe4c410139191d3d6383451ecf89ab4da969d9edeb95b4274e34495ad26"
Nov 02 02:17:27.421177 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:17:27.421138    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:17:32.914566 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:32.914531    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:17:39.218246 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:39.218205    5639 scope.go:122] "RemoveContainer" containerID="844ba007e78ce4ad32234aeae1adbb5b91c1f7914aa6ac48729a4bf873a2b2b7"
Nov 02 02:17:39.228081 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:39.228058    5639 scope.go:122] "RemoveContainer" containerID="237ff5de638a6d6f327f60a1e7f7f6f43a284f5ce70dbe286a8564e3146f77d6"
Nov 02 02:17:39.236848 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:39.236824    5639 scope.go:122] "RemoveContainer" containerID="a29520a468861a3629622900e57eea8da284fa66ad4cc0409ee70db163086a29"
Nov 02 02:17:39.245620 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:39.245593    5639 scope.go:122] "RemoveContainer" containerID="de30ae11b86ca31778506f6d1a4a87755237204eeb7e4c3c91254466911cff91"
Nov 02 02:17:39.254165 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:39.254144    5639 scope.go:122] "RemoveContainer" containerID="1650be9706249db165f5905e674a23e38d4e56658fcdbbb86562b4b7917ee456"
Nov 02 02:17:39.262559 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:39.262540    5639 scope.go:122] "RemoveContainer" containerID="0f1f770d8206e8c9de8ed4440152f237aed9ef54b24ccfe6b00e60d16c8434bf"
Nov 02 02:17:51.139776 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:51.139736    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nkhnx"]
Nov 02 02:17:51.140211 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:51.140195    5639 cpu_manager.go:414] "containerMap: removing container" podUID="e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4" containerName="revoke-keys"
Nov 02 02:17:51.140268 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:51.140213    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4" containerName="revoke-keys"
Nov 02 02:17:51.140329 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:51.140308    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4" containerName="revoke-keys"
Nov 02 02:17:51.145427 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:51.145399    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nkhnx"
Nov 02 02:17:51.185838 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:51.185803    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22242546-a0a6-4a08-872b-0e3cd4d57689-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-nkhnx\" (UID: \"22242546-a0a6-4a08-872b-0e3cd4d57689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nkhnx"
Nov 02 02:17:51.186020 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:51.185881    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22242546-a0a6-4a08-872b-0e3cd4d57689-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-nkhnx\" (UID: \"22242546-a0a6-4a08-872b-0e3cd4d57689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nkhnx"
Nov 02 02:17:51.186071 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:51.186037    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6msn\" (UniqueName: \"kubernetes.io/projected/22242546-a0a6-4a08-872b-0e3cd4d57689-kube-api-access-m6msn\") pod \"maas-api-revoke-sub-046e1343-nkhnx\" (UID: \"22242546-a0a6-4a08-872b-0e3cd4d57689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nkhnx"
Nov 02 02:17:51.198192 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:51.198150    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:17:51.200387 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:51.200355    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nkhnx"]
Nov 02 02:17:51.247747 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:51.247705    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:17:51.247747 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:51.247724    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:17:51.247965 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:51.247821    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:17:51.287165 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:51.287127    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6msn\" (UniqueName: \"kubernetes.io/projected/22242546-a0a6-4a08-872b-0e3cd4d57689-kube-api-access-m6msn\") pod \"maas-api-revoke-sub-046e1343-nkhnx\" (UID: \"22242546-a0a6-4a08-872b-0e3cd4d57689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nkhnx"
Nov 02 02:17:51.287403 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:51.287209    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22242546-a0a6-4a08-872b-0e3cd4d57689-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-nkhnx\" (UID: \"22242546-a0a6-4a08-872b-0e3cd4d57689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nkhnx"
Nov 02 02:17:51.287403 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:51.287263    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22242546-a0a6-4a08-872b-0e3cd4d57689-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-nkhnx\" (UID: \"22242546-a0a6-4a08-872b-0e3cd4d57689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nkhnx"
Nov 02 02:17:51.287928 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:51.287908    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22242546-a0a6-4a08-872b-0e3cd4d57689-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-nkhnx\" (UID: \"22242546-a0a6-4a08-872b-0e3cd4d57689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nkhnx"
Nov 02 02:17:51.289750 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:51.289727    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22242546-a0a6-4a08-872b-0e3cd4d57689-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-nkhnx\" (UID: \"22242546-a0a6-4a08-872b-0e3cd4d57689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nkhnx"
Nov 02 02:17:51.348110 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:51.348078    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6msn\" (UniqueName: \"kubernetes.io/projected/22242546-a0a6-4a08-872b-0e3cd4d57689-kube-api-access-m6msn\") pod \"maas-api-revoke-sub-046e1343-nkhnx\" (UID: \"22242546-a0a6-4a08-872b-0e3cd4d57689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nkhnx"
Nov 02 02:17:51.459681 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:51.459587    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nkhnx"
Nov 02 02:17:51.833428 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:51.833398    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nkhnx"]
Nov 02 02:17:51.835953 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:17:51.835925    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22242546_a0a6_4a08_872b_0e3cd4d57689.slice/crio-f99d845ef34c41c585e8334a71764ac9a83197a3533b210374fe5a0ad1833a6f WatchSource:0}: Error finding container f99d845ef34c41c585e8334a71764ac9a83197a3533b210374fe5a0ad1833a6f: Status 404 returned error can't find the container with id f99d845ef34c41c585e8334a71764ac9a83197a3533b210374fe5a0ad1833a6f
Nov 02 02:17:52.023358 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:52.023311    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nkhnx" event={"ID":"22242546-a0a6-4a08-872b-0e3cd4d57689","Type":"ContainerStarted","Data":"f99d845ef34c41c585e8334a71764ac9a83197a3533b210374fe5a0ad1833a6f"}
Nov 02 02:17:53.030403 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:53.030273    5639 generic.go:356] "Generic (PLEG): container finished" podID="22242546-a0a6-4a08-872b-0e3cd4d57689" containerID="8150bba3eb1322a04d9b4a160de7ffff45b88491e836d6928040967cc5167f34" exitCode=0
Nov 02 02:17:53.030403 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:53.030359    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nkhnx" event={"ID":"22242546-a0a6-4a08-872b-0e3cd4d57689","Type":"ContainerDied","Data":"8150bba3eb1322a04d9b4a160de7ffff45b88491e836d6928040967cc5167f34"}
Nov 02 02:17:54.176746 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:54.176716    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nkhnx"
Nov 02 02:17:54.317893 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:54.317795    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22242546-a0a6-4a08-872b-0e3cd4d57689-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22242546-a0a6-4a08-872b-0e3cd4d57689-maas-api-xks-ca\") pod \"22242546-a0a6-4a08-872b-0e3cd4d57689\" (UID: \"22242546-a0a6-4a08-872b-0e3cd4d57689\") "
Nov 02 02:17:54.317893 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:54.317890    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22242546-a0a6-4a08-872b-0e3cd4d57689-kube-api-access-m6msn\" (UniqueName: \"kubernetes.io/projected/22242546-a0a6-4a08-872b-0e3cd4d57689-kube-api-access-m6msn\") pod \"22242546-a0a6-4a08-872b-0e3cd4d57689\" (UID: \"22242546-a0a6-4a08-872b-0e3cd4d57689\") "
Nov 02 02:17:54.318148 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:54.317995    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22242546-a0a6-4a08-872b-0e3cd4d57689-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22242546-a0a6-4a08-872b-0e3cd4d57689-maas-api-openshift-ca\") pod \"22242546-a0a6-4a08-872b-0e3cd4d57689\" (UID: \"22242546-a0a6-4a08-872b-0e3cd4d57689\") "
Nov 02 02:17:54.318445 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:54.318406    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22242546-a0a6-4a08-872b-0e3cd4d57689-maas-api-openshift-ca" pod "22242546-a0a6-4a08-872b-0e3cd4d57689" (UID: "22242546-a0a6-4a08-872b-0e3cd4d57689"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:17:54.320168 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:54.320138    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22242546-a0a6-4a08-872b-0e3cd4d57689-maas-api-xks-ca" pod "22242546-a0a6-4a08-872b-0e3cd4d57689" (UID: "22242546-a0a6-4a08-872b-0e3cd4d57689"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:17:54.320709 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:54.320682    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22242546-a0a6-4a08-872b-0e3cd4d57689-kube-api-access-m6msn" pod "22242546-a0a6-4a08-872b-0e3cd4d57689" (UID: "22242546-a0a6-4a08-872b-0e3cd4d57689"). InnerVolumeSpecName "kube-api-access-m6msn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:17:54.419791 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:54.419754    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22242546-a0a6-4a08-872b-0e3cd4d57689-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:54.419791 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:54.419787    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22242546-a0a6-4a08-872b-0e3cd4d57689-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:54.419791 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:54.419798    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6msn\" (UniqueName: \"kubernetes.io/projected/22242546-a0a6-4a08-872b-0e3cd4d57689-kube-api-access-m6msn\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:55.041723 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:55.041687    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nkhnx"
Nov 02 02:17:55.041723 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:55.041710    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nkhnx" event={"ID":"22242546-a0a6-4a08-872b-0e3cd4d57689","Type":"ContainerDied","Data":"f99d845ef34c41c585e8334a71764ac9a83197a3533b210374fe5a0ad1833a6f"}
Nov 02 02:17:55.041961 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:17:55.041762    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f99d845ef34c41c585e8334a71764ac9a83197a3533b210374fe5a0ad1833a6f"
Nov 02 02:17:57.420619 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:17:57.420581    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:18:01.903093 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:01.903045    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n7xh7"]
Nov 02 02:18:01.903844 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:01.903821    5639 cpu_manager.go:414] "containerMap: removing container" podUID="22242546-a0a6-4a08-872b-0e3cd4d57689" containerName="revoke-keys"
Nov 02 02:18:01.903897 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:01.903848    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22242546-a0a6-4a08-872b-0e3cd4d57689" containerName="revoke-keys"
Nov 02 02:18:01.904016 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:01.904004    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="22242546-a0a6-4a08-872b-0e3cd4d57689" containerName="revoke-keys"
Nov 02 02:18:01.907776 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:01.907757    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n7xh7"
Nov 02 02:18:01.963030 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:01.962989    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n7xh7"]
Nov 02 02:18:02.004160 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:02.004123    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpwdj\" (UniqueName: \"kubernetes.io/projected/40123de6-f019-4408-bed9-8685479a3ff0-kube-api-access-fpwdj\") pod \"maas-api-revoke-sub-d3fba056-n7xh7\" (UID: \"40123de6-f019-4408-bed9-8685479a3ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n7xh7"
Nov 02 02:18:02.004160 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:02.004177    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40123de6-f019-4408-bed9-8685479a3ff0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-n7xh7\" (UID: \"40123de6-f019-4408-bed9-8685479a3ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n7xh7"
Nov 02 02:18:02.004414 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:02.004344    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40123de6-f019-4408-bed9-8685479a3ff0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-n7xh7\" (UID: \"40123de6-f019-4408-bed9-8685479a3ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n7xh7"
Nov 02 02:18:02.009840 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:02.009812    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:18:02.009973 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:02.009846    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:18:02.009973 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:02.009847    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:18:02.009973 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:02.009872    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:18:02.105465 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:02.105433    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpwdj\" (UniqueName: \"kubernetes.io/projected/40123de6-f019-4408-bed9-8685479a3ff0-kube-api-access-fpwdj\") pod \"maas-api-revoke-sub-d3fba056-n7xh7\" (UID: \"40123de6-f019-4408-bed9-8685479a3ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n7xh7"
Nov 02 02:18:02.105677 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:02.105481    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40123de6-f019-4408-bed9-8685479a3ff0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-n7xh7\" (UID: \"40123de6-f019-4408-bed9-8685479a3ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n7xh7"
Nov 02 02:18:02.105677 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:02.105535    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40123de6-f019-4408-bed9-8685479a3ff0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-n7xh7\" (UID: \"40123de6-f019-4408-bed9-8685479a3ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n7xh7"
Nov 02 02:18:02.106202 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:02.106181    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40123de6-f019-4408-bed9-8685479a3ff0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-n7xh7\" (UID: \"40123de6-f019-4408-bed9-8685479a3ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n7xh7"
Nov 02 02:18:02.108197 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:02.108174    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40123de6-f019-4408-bed9-8685479a3ff0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-n7xh7\" (UID: \"40123de6-f019-4408-bed9-8685479a3ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n7xh7"
Nov 02 02:18:02.168835 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:02.168767    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpwdj\" (UniqueName: \"kubernetes.io/projected/40123de6-f019-4408-bed9-8685479a3ff0-kube-api-access-fpwdj\") pod \"maas-api-revoke-sub-d3fba056-n7xh7\" (UID: \"40123de6-f019-4408-bed9-8685479a3ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n7xh7"
Nov 02 02:18:02.225010 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:02.224970    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n7xh7"
Nov 02 02:18:02.599722 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:02.599697    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n7xh7"]
Nov 02 02:18:02.602105 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:18:02.602078    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40123de6_f019_4408_bed9_8685479a3ff0.slice/crio-5c4d63080e2112b79f219a9cc3c7ebd19d46a6b92935e1802b995f0d6c8e5af6 WatchSource:0}: Error finding container 5c4d63080e2112b79f219a9cc3c7ebd19d46a6b92935e1802b995f0d6c8e5af6: Status 404 returned error can't find the container with id 5c4d63080e2112b79f219a9cc3c7ebd19d46a6b92935e1802b995f0d6c8e5af6
Nov 02 02:18:03.087975 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:03.087931    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n7xh7" event={"ID":"40123de6-f019-4408-bed9-8685479a3ff0","Type":"ContainerStarted","Data":"9f66d2b9a3a6a45f2cb79206b2976b0427677df60dd69c0cf9b907da71310c3c"}
Nov 02 02:18:03.088520 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:03.087990    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n7xh7" event={"ID":"40123de6-f019-4408-bed9-8685479a3ff0","Type":"ContainerStarted","Data":"5c4d63080e2112b79f219a9cc3c7ebd19d46a6b92935e1802b995f0d6c8e5af6"}
Nov 02 02:18:03.204177 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:03.204055    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n7xh7" podStartSLOduration=2.204039714 podStartE2EDuration="2.204039714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:18:03.202878521 +0000 UTC m=+3148.106560639" watchObservedRunningTime="2026-11-02 02:18:03.204039714 +0000 UTC m=+3148.107721828"
Nov 02 02:18:04.094478 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:04.094396    5639 generic.go:356] "Generic (PLEG): container finished" podID="40123de6-f019-4408-bed9-8685479a3ff0" containerID="9f66d2b9a3a6a45f2cb79206b2976b0427677df60dd69c0cf9b907da71310c3c" exitCode=0
Nov 02 02:18:04.094478 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:04.094464    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n7xh7" event={"ID":"40123de6-f019-4408-bed9-8685479a3ff0","Type":"ContainerDied","Data":"9f66d2b9a3a6a45f2cb79206b2976b0427677df60dd69c0cf9b907da71310c3c"}
Nov 02 02:18:04.101302 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:04.101246    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:18:05.309170 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:05.309132    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n7xh7"
Nov 02 02:18:05.448891 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:05.448784    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40123de6-f019-4408-bed9-8685479a3ff0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40123de6-f019-4408-bed9-8685479a3ff0-maas-api-xks-ca\") pod \"40123de6-f019-4408-bed9-8685479a3ff0\" (UID: \"40123de6-f019-4408-bed9-8685479a3ff0\") "
Nov 02 02:18:05.449230 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:05.449209    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40123de6-f019-4408-bed9-8685479a3ff0-kube-api-access-fpwdj\" (UniqueName: \"kubernetes.io/projected/40123de6-f019-4408-bed9-8685479a3ff0-kube-api-access-fpwdj\") pod \"40123de6-f019-4408-bed9-8685479a3ff0\" (UID: \"40123de6-f019-4408-bed9-8685479a3ff0\") "
Nov 02 02:18:05.449425 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:05.449406    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40123de6-f019-4408-bed9-8685479a3ff0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40123de6-f019-4408-bed9-8685479a3ff0-maas-api-openshift-ca\") pod \"40123de6-f019-4408-bed9-8685479a3ff0\" (UID: \"40123de6-f019-4408-bed9-8685479a3ff0\") "
Nov 02 02:18:05.450585 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:05.450555    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40123de6-f019-4408-bed9-8685479a3ff0-maas-api-openshift-ca" pod "40123de6-f019-4408-bed9-8685479a3ff0" (UID: "40123de6-f019-4408-bed9-8685479a3ff0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:18:05.486426 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:05.486381    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40123de6-f019-4408-bed9-8685479a3ff0-kube-api-access-fpwdj" pod "40123de6-f019-4408-bed9-8685479a3ff0" (UID: "40123de6-f019-4408-bed9-8685479a3ff0"). InnerVolumeSpecName "kube-api-access-fpwdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:18:05.499897 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:05.499858    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40123de6-f019-4408-bed9-8685479a3ff0-maas-api-xks-ca" pod "40123de6-f019-4408-bed9-8685479a3ff0" (UID: "40123de6-f019-4408-bed9-8685479a3ff0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:18:05.603116 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:05.603074    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40123de6-f019-4408-bed9-8685479a3ff0-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:18:05.603415 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:05.603391    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpwdj\" (UniqueName: \"kubernetes.io/projected/40123de6-f019-4408-bed9-8685479a3ff0-kube-api-access-fpwdj\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:18:05.603578 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:05.603562    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40123de6-f019-4408-bed9-8685479a3ff0-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:18:06.142307 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:06.142228    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n7xh7" event={"ID":"40123de6-f019-4408-bed9-8685479a3ff0","Type":"ContainerDied","Data":"5c4d63080e2112b79f219a9cc3c7ebd19d46a6b92935e1802b995f0d6c8e5af6"}
Nov 02 02:18:06.142546 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:06.142406    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c4d63080e2112b79f219a9cc3c7ebd19d46a6b92935e1802b995f0d6c8e5af6"
Nov 02 02:18:06.142546 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:06.142491    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n7xh7"
Nov 02 02:18:14.686962 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:14.686925    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:18:27.114480 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:27.114398    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:18:27.420987 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:18:27.420894    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:18:36.656951 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:36.656912    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p66fd"]
Nov 02 02:18:36.657411 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:36.657385    5639 cpu_manager.go:414] "containerMap: removing container" podUID="40123de6-f019-4408-bed9-8685479a3ff0" containerName="revoke-keys"
Nov 02 02:18:36.657411 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:36.657398    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40123de6-f019-4408-bed9-8685479a3ff0" containerName="revoke-keys"
Nov 02 02:18:36.657513 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:36.657464    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="40123de6-f019-4408-bed9-8685479a3ff0" containerName="revoke-keys"
Nov 02 02:18:36.660743 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:36.660721    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p66fd"
Nov 02 02:18:36.715108 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:36.715075    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:18:36.717730 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:36.717695    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p66fd"]
Nov 02 02:18:36.764631 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:36.764593    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:18:36.764631 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:36.764593    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:18:36.764852 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:36.764663    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:18:36.826256 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:36.826217    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9pjg\" (UniqueName: \"kubernetes.io/projected/83668079-47fe-4dd4-82d1-227ea5b13c31-kube-api-access-q9pjg\") pod \"maas-api-revoke-sub-ae594979-p66fd\" (UID: \"83668079-47fe-4dd4-82d1-227ea5b13c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p66fd"
Nov 02 02:18:36.826932 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:36.826372    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83668079-47fe-4dd4-82d1-227ea5b13c31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-p66fd\" (UID: \"83668079-47fe-4dd4-82d1-227ea5b13c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p66fd"
Nov 02 02:18:36.826932 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:36.826413    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83668079-47fe-4dd4-82d1-227ea5b13c31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-p66fd\" (UID: \"83668079-47fe-4dd4-82d1-227ea5b13c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p66fd"
Nov 02 02:18:36.927951 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:36.927854    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9pjg\" (UniqueName: \"kubernetes.io/projected/83668079-47fe-4dd4-82d1-227ea5b13c31-kube-api-access-q9pjg\") pod \"maas-api-revoke-sub-ae594979-p66fd\" (UID: \"83668079-47fe-4dd4-82d1-227ea5b13c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p66fd"
Nov 02 02:18:36.927951 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:36.927930    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83668079-47fe-4dd4-82d1-227ea5b13c31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-p66fd\" (UID: \"83668079-47fe-4dd4-82d1-227ea5b13c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p66fd"
Nov 02 02:18:36.927951 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:36.927949    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83668079-47fe-4dd4-82d1-227ea5b13c31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-p66fd\" (UID: \"83668079-47fe-4dd4-82d1-227ea5b13c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p66fd"
Nov 02 02:18:36.928589 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:36.928571    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83668079-47fe-4dd4-82d1-227ea5b13c31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-p66fd\" (UID: \"83668079-47fe-4dd4-82d1-227ea5b13c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p66fd"
Nov 02 02:18:36.930573 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:36.930551    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83668079-47fe-4dd4-82d1-227ea5b13c31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-p66fd\" (UID: \"83668079-47fe-4dd4-82d1-227ea5b13c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p66fd"
Nov 02 02:18:36.988879 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:36.988833    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9pjg\" (UniqueName: \"kubernetes.io/projected/83668079-47fe-4dd4-82d1-227ea5b13c31-kube-api-access-q9pjg\") pod \"maas-api-revoke-sub-ae594979-p66fd\" (UID: \"83668079-47fe-4dd4-82d1-227ea5b13c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p66fd"
Nov 02 02:18:37.274529 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:37.274447    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p66fd"
Nov 02 02:18:37.647942 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:37.647917    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p66fd"]
Nov 02 02:18:37.650378 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:18:37.650337    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83668079_47fe_4dd4_82d1_227ea5b13c31.slice/crio-2bf9bd637d1a2be4513a15afd685748a3f90a0a78f140168f754aee827d53e95 WatchSource:0}: Error finding container 2bf9bd637d1a2be4513a15afd685748a3f90a0a78f140168f754aee827d53e95: Status 404 returned error can't find the container with id 2bf9bd637d1a2be4513a15afd685748a3f90a0a78f140168f754aee827d53e95
Nov 02 02:18:38.356497 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:38.356463    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p66fd" event={"ID":"83668079-47fe-4dd4-82d1-227ea5b13c31","Type":"ContainerStarted","Data":"d5aaaa83ba7c735b1ac0efcc02db9778ec7d36e7bc02e3d4367f79c6f6dc879b"}
Nov 02 02:18:38.356884 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:38.356515    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p66fd" event={"ID":"83668079-47fe-4dd4-82d1-227ea5b13c31","Type":"ContainerStarted","Data":"2bf9bd637d1a2be4513a15afd685748a3f90a0a78f140168f754aee827d53e95"}
Nov 02 02:18:38.472502 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:38.472433    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p66fd" podStartSLOduration=2.47241772 podStartE2EDuration="2.47241772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:18:38.472090862 +0000 UTC m=+3183.375772978" watchObservedRunningTime="2026-11-02 02:18:38.47241772 +0000 UTC m=+3183.376099832"
Nov 02 02:18:39.304985 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:39.304936    5639 scope.go:122] "RemoveContainer" containerID="da713d436c5490b5e1990e1f40edcdc40d4113a2775cd2ed615dd6ed97bf2d2d"
Nov 02 02:18:39.314153 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:39.314120    5639 scope.go:122] "RemoveContainer" containerID="a8a04ef230adbedb2f4cbec71b54181a1bc30e4feb3566fc43dfc1f206cfbc93"
Nov 02 02:18:39.362569 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:39.362533    5639 generic.go:356] "Generic (PLEG): container finished" podID="83668079-47fe-4dd4-82d1-227ea5b13c31" containerID="d5aaaa83ba7c735b1ac0efcc02db9778ec7d36e7bc02e3d4367f79c6f6dc879b" exitCode=0
Nov 02 02:18:39.362978 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:39.362625    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p66fd" event={"ID":"83668079-47fe-4dd4-82d1-227ea5b13c31","Type":"ContainerDied","Data":"d5aaaa83ba7c735b1ac0efcc02db9778ec7d36e7bc02e3d4367f79c6f6dc879b"}
Nov 02 02:18:40.513693 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:40.513666    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p66fd"
Nov 02 02:18:40.563002 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:40.562964    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83668079-47fe-4dd4-82d1-227ea5b13c31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83668079-47fe-4dd4-82d1-227ea5b13c31-maas-api-xks-ca\") pod \"83668079-47fe-4dd4-82d1-227ea5b13c31\" (UID: \"83668079-47fe-4dd4-82d1-227ea5b13c31\") "
Nov 02 02:18:40.563141 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:40.563083    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83668079-47fe-4dd4-82d1-227ea5b13c31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83668079-47fe-4dd4-82d1-227ea5b13c31-maas-api-openshift-ca\") pod \"83668079-47fe-4dd4-82d1-227ea5b13c31\" (UID: \"83668079-47fe-4dd4-82d1-227ea5b13c31\") "
Nov 02 02:18:40.563141 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:40.563101    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83668079-47fe-4dd4-82d1-227ea5b13c31-kube-api-access-q9pjg\" (UniqueName: \"kubernetes.io/projected/83668079-47fe-4dd4-82d1-227ea5b13c31-kube-api-access-q9pjg\") pod \"83668079-47fe-4dd4-82d1-227ea5b13c31\" (UID: \"83668079-47fe-4dd4-82d1-227ea5b13c31\") "
Nov 02 02:18:40.563462 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:40.563436    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83668079-47fe-4dd4-82d1-227ea5b13c31-maas-api-openshift-ca" pod "83668079-47fe-4dd4-82d1-227ea5b13c31" (UID: "83668079-47fe-4dd4-82d1-227ea5b13c31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:18:40.565231 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:40.565209    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83668079-47fe-4dd4-82d1-227ea5b13c31-maas-api-xks-ca" pod "83668079-47fe-4dd4-82d1-227ea5b13c31" (UID: "83668079-47fe-4dd4-82d1-227ea5b13c31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:18:40.565330 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:40.565266    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83668079-47fe-4dd4-82d1-227ea5b13c31-kube-api-access-q9pjg" pod "83668079-47fe-4dd4-82d1-227ea5b13c31" (UID: "83668079-47fe-4dd4-82d1-227ea5b13c31"). InnerVolumeSpecName "kube-api-access-q9pjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:18:40.664184 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:40.664084    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83668079-47fe-4dd4-82d1-227ea5b13c31-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:18:40.664184 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:40.664129    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83668079-47fe-4dd4-82d1-227ea5b13c31-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:18:40.664184 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:40.664139    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9pjg\" (UniqueName: \"kubernetes.io/projected/83668079-47fe-4dd4-82d1-227ea5b13c31-kube-api-access-q9pjg\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:18:41.381383 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:41.381345    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p66fd" event={"ID":"83668079-47fe-4dd4-82d1-227ea5b13c31","Type":"ContainerDied","Data":"2bf9bd637d1a2be4513a15afd685748a3f90a0a78f140168f754aee827d53e95"}
Nov 02 02:18:41.381569 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:41.381415    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p66fd"
Nov 02 02:18:41.381569 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:41.381423    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bf9bd637d1a2be4513a15afd685748a3f90a0a78f140168f754aee827d53e95"
Nov 02 02:18:49.506445 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:49.506405    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:18:57.437872 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:18:57.437794    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:18:59.114437 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:18:59.114397    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:19:04.629189 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:04.629150    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lgm9s"]
Nov 02 02:19:04.629646 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:04.629616    5639 cpu_manager.go:414] "containerMap: removing container" podUID="83668079-47fe-4dd4-82d1-227ea5b13c31" containerName="revoke-keys"
Nov 02 02:19:04.629646 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:04.629630    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83668079-47fe-4dd4-82d1-227ea5b13c31" containerName="revoke-keys"
Nov 02 02:19:04.629747 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:04.629718    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="83668079-47fe-4dd4-82d1-227ea5b13c31" containerName="revoke-keys"
Nov 02 02:19:04.633144 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:04.633112    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lgm9s"
Nov 02 02:19:04.687544 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:04.687507    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:19:04.736900 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:04.736853    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:19:04.736900 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:04.736861    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:19:04.736900 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:04.736896    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:19:04.739073 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:04.739045    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lgm9s"]
Nov 02 02:19:04.750546 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:04.750517    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1301372-aa12-416f-a565-74989f5641ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-lgm9s\" (UID: \"f1301372-aa12-416f-a565-74989f5641ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lgm9s"
Nov 02 02:19:04.750692 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:04.750548    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b2nl\" (UniqueName: \"kubernetes.io/projected/f1301372-aa12-416f-a565-74989f5641ff-kube-api-access-9b2nl\") pod \"maas-api-revoke-sub-a4aef7d7-lgm9s\" (UID: \"f1301372-aa12-416f-a565-74989f5641ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lgm9s"
Nov 02 02:19:04.750692 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:04.750635    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1301372-aa12-416f-a565-74989f5641ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-lgm9s\" (UID: \"f1301372-aa12-416f-a565-74989f5641ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lgm9s"
Nov 02 02:19:04.851657 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:04.851617    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1301372-aa12-416f-a565-74989f5641ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-lgm9s\" (UID: \"f1301372-aa12-416f-a565-74989f5641ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lgm9s"
Nov 02 02:19:04.851869 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:04.851725    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1301372-aa12-416f-a565-74989f5641ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-lgm9s\" (UID: \"f1301372-aa12-416f-a565-74989f5641ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lgm9s"
Nov 02 02:19:04.851869 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:04.851748    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b2nl\" (UniqueName: \"kubernetes.io/projected/f1301372-aa12-416f-a565-74989f5641ff-kube-api-access-9b2nl\") pod \"maas-api-revoke-sub-a4aef7d7-lgm9s\" (UID: \"f1301372-aa12-416f-a565-74989f5641ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lgm9s"
Nov 02 02:19:04.852532 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:04.852505    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1301372-aa12-416f-a565-74989f5641ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-lgm9s\" (UID: \"f1301372-aa12-416f-a565-74989f5641ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lgm9s"
Nov 02 02:19:04.854420 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:04.854396    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1301372-aa12-416f-a565-74989f5641ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-lgm9s\" (UID: \"f1301372-aa12-416f-a565-74989f5641ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lgm9s"
Nov 02 02:19:04.913522 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:04.913439    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b2nl\" (UniqueName: \"kubernetes.io/projected/f1301372-aa12-416f-a565-74989f5641ff-kube-api-access-9b2nl\") pod \"maas-api-revoke-sub-a4aef7d7-lgm9s\" (UID: \"f1301372-aa12-416f-a565-74989f5641ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lgm9s"
Nov 02 02:19:04.946073 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:04.946039    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lgm9s"
Nov 02 02:19:05.321527 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:05.321495    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lgm9s"]
Nov 02 02:19:05.323484 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:19:05.323450    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1301372_aa12_416f_a565_74989f5641ff.slice/crio-935e449f398e02bd4c04fbe4a6a1f53de464f67e1afe457463d7c3186f5c853f WatchSource:0}: Error finding container 935e449f398e02bd4c04fbe4a6a1f53de464f67e1afe457463d7c3186f5c853f: Status 404 returned error can't find the container with id 935e449f398e02bd4c04fbe4a6a1f53de464f67e1afe457463d7c3186f5c853f
Nov 02 02:19:05.521566 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:05.521517    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lgm9s" event={"ID":"f1301372-aa12-416f-a565-74989f5641ff","Type":"ContainerStarted","Data":"935e449f398e02bd4c04fbe4a6a1f53de464f67e1afe457463d7c3186f5c853f"}
Nov 02 02:19:06.528502 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:06.528400    5639 generic.go:356] "Generic (PLEG): container finished" podID="f1301372-aa12-416f-a565-74989f5641ff" containerID="e037fa688ced19da343b59ca11dbc07a611697fd6b9befbc4d4df61682c58220" exitCode=0
Nov 02 02:19:06.528502 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:06.528443    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lgm9s" event={"ID":"f1301372-aa12-416f-a565-74989f5641ff","Type":"ContainerDied","Data":"e037fa688ced19da343b59ca11dbc07a611697fd6b9befbc4d4df61682c58220"}
Nov 02 02:19:07.675058 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:07.675031    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lgm9s"
Nov 02 02:19:07.778141 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:07.778097    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1301372-aa12-416f-a565-74989f5641ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1301372-aa12-416f-a565-74989f5641ff-maas-api-xks-ca\") pod \"f1301372-aa12-416f-a565-74989f5641ff\" (UID: \"f1301372-aa12-416f-a565-74989f5641ff\") "
Nov 02 02:19:07.778387 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:07.778214    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1301372-aa12-416f-a565-74989f5641ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1301372-aa12-416f-a565-74989f5641ff-maas-api-openshift-ca\") pod \"f1301372-aa12-416f-a565-74989f5641ff\" (UID: \"f1301372-aa12-416f-a565-74989f5641ff\") "
Nov 02 02:19:07.778387 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:07.778364    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1301372-aa12-416f-a565-74989f5641ff-kube-api-access-9b2nl\" (UniqueName: \"kubernetes.io/projected/f1301372-aa12-416f-a565-74989f5641ff-kube-api-access-9b2nl\") pod \"f1301372-aa12-416f-a565-74989f5641ff\" (UID: \"f1301372-aa12-416f-a565-74989f5641ff\") "
Nov 02 02:19:07.778653 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:07.778628    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1301372-aa12-416f-a565-74989f5641ff-maas-api-openshift-ca" pod "f1301372-aa12-416f-a565-74989f5641ff" (UID: "f1301372-aa12-416f-a565-74989f5641ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:19:07.780427 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:07.780403    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1301372-aa12-416f-a565-74989f5641ff-maas-api-xks-ca" pod "f1301372-aa12-416f-a565-74989f5641ff" (UID: "f1301372-aa12-416f-a565-74989f5641ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:19:07.780598 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:07.780575    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1301372-aa12-416f-a565-74989f5641ff-kube-api-access-9b2nl" pod "f1301372-aa12-416f-a565-74989f5641ff" (UID: "f1301372-aa12-416f-a565-74989f5641ff"). InnerVolumeSpecName "kube-api-access-9b2nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:19:07.880137 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:07.880097    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1301372-aa12-416f-a565-74989f5641ff-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:19:07.880137 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:07.880130    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1301372-aa12-416f-a565-74989f5641ff-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:19:07.880137 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:07.880141    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b2nl\" (UniqueName: \"kubernetes.io/projected/f1301372-aa12-416f-a565-74989f5641ff-kube-api-access-9b2nl\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:19:08.540158 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:08.540129    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lgm9s"
Nov 02 02:19:08.540384 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:08.540173    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lgm9s" event={"ID":"f1301372-aa12-416f-a565-74989f5641ff","Type":"ContainerDied","Data":"935e449f398e02bd4c04fbe4a6a1f53de464f67e1afe457463d7c3186f5c853f"}
Nov 02 02:19:08.540384 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:08.540225    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="935e449f398e02bd4c04fbe4a6a1f53de464f67e1afe457463d7c3186f5c853f"
Nov 02 02:19:17.492267 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:17.492227    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:19:27.421005 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:19:27.420963    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:19:29.803042 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:29.803005    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:19:32.060069 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:32.060029    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5064232b-rwcz5"]
Nov 02 02:19:32.109402 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:32.109367    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5064232b-rwcz5"]
Nov 02 02:19:33.218081 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:33.218045    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dcecf9b-caa8-4a47-bc54-7e4d1302c843" path="/var/lib/kubelet/pods/0dcecf9b-caa8-4a47-bc54-7e4d1302c843/volumes"
Nov 02 02:19:35.354934 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:35.354902    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lqjpk"]
Nov 02 02:19:35.355443 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:35.355428    5639 cpu_manager.go:414] "containerMap: removing container" podUID="f1301372-aa12-416f-a565-74989f5641ff" containerName="revoke-keys"
Nov 02 02:19:35.355497 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:35.355445    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1301372-aa12-416f-a565-74989f5641ff" containerName="revoke-keys"
Nov 02 02:19:35.355553 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:35.355542    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1301372-aa12-416f-a565-74989f5641ff" containerName="revoke-keys"
Nov 02 02:19:35.359524 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:35.359504    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lqjpk"
Nov 02 02:19:35.412003 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:35.411974    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:19:35.426045 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:35.426003    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2jb8\" (UniqueName: \"kubernetes.io/projected/08ef4ca8-74b3-4d03-b24d-2d421d42edf4-kube-api-access-x2jb8\") pod \"maas-api-revoke-sub-79805ca9-lqjpk\" (UID: \"08ef4ca8-74b3-4d03-b24d-2d421d42edf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lqjpk"
Nov 02 02:19:35.426241 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:35.426123    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ef4ca8-74b3-4d03-b24d-2d421d42edf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-lqjpk\" (UID: \"08ef4ca8-74b3-4d03-b24d-2d421d42edf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lqjpk"
Nov 02 02:19:35.426241 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:35.426194    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ef4ca8-74b3-4d03-b24d-2d421d42edf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-lqjpk\" (UID: \"08ef4ca8-74b3-4d03-b24d-2d421d42edf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lqjpk"
Nov 02 02:19:35.461598 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:35.461563    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:19:35.461598 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:35.461596    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:19:35.461854 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:35.461655    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:19:35.463734 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:35.463716    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lqjpk"]
Nov 02 02:19:35.527577 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:35.527534    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2jb8\" (UniqueName: \"kubernetes.io/projected/08ef4ca8-74b3-4d03-b24d-2d421d42edf4-kube-api-access-x2jb8\") pod \"maas-api-revoke-sub-79805ca9-lqjpk\" (UID: \"08ef4ca8-74b3-4d03-b24d-2d421d42edf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lqjpk"
Nov 02 02:19:35.527762 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:35.527589    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ef4ca8-74b3-4d03-b24d-2d421d42edf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-lqjpk\" (UID: \"08ef4ca8-74b3-4d03-b24d-2d421d42edf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lqjpk"
Nov 02 02:19:35.527762 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:35.527640    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ef4ca8-74b3-4d03-b24d-2d421d42edf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-lqjpk\" (UID: \"08ef4ca8-74b3-4d03-b24d-2d421d42edf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lqjpk"
Nov 02 02:19:35.528419 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:35.528400    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ef4ca8-74b3-4d03-b24d-2d421d42edf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-lqjpk\" (UID: \"08ef4ca8-74b3-4d03-b24d-2d421d42edf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lqjpk"
Nov 02 02:19:35.530168 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:35.530150    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ef4ca8-74b3-4d03-b24d-2d421d42edf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-lqjpk\" (UID: \"08ef4ca8-74b3-4d03-b24d-2d421d42edf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lqjpk"
Nov 02 02:19:35.589155 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:35.589119    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2jb8\" (UniqueName: \"kubernetes.io/projected/08ef4ca8-74b3-4d03-b24d-2d421d42edf4-kube-api-access-x2jb8\") pod \"maas-api-revoke-sub-79805ca9-lqjpk\" (UID: \"08ef4ca8-74b3-4d03-b24d-2d421d42edf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lqjpk"
Nov 02 02:19:35.672480 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:35.672373    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lqjpk"
Nov 02 02:19:36.047003 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:36.046975    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lqjpk"]
Nov 02 02:19:36.048815 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:19:36.048782    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ef4ca8_74b3_4d03_b24d_2d421d42edf4.slice/crio-06c3877df214529abedcb57d011cffc97290485f482db202862e4f38adf40927 WatchSource:0}: Error finding container 06c3877df214529abedcb57d011cffc97290485f482db202862e4f38adf40927: Status 404 returned error can't find the container with id 06c3877df214529abedcb57d011cffc97290485f482db202862e4f38adf40927
Nov 02 02:19:36.721649 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:36.721550    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lqjpk" event={"ID":"08ef4ca8-74b3-4d03-b24d-2d421d42edf4","Type":"ContainerStarted","Data":"99b29efa271caadfc9b614c5e8ed8b60ad3a5c25490b5b57d5c5b7a736180e87"}
Nov 02 02:19:36.721649 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:36.721617    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lqjpk" event={"ID":"08ef4ca8-74b3-4d03-b24d-2d421d42edf4","Type":"ContainerStarted","Data":"06c3877df214529abedcb57d011cffc97290485f482db202862e4f38adf40927"}
Nov 02 02:19:36.838235 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:36.838184    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lqjpk" podStartSLOduration=1.8381688550000002 podStartE2EDuration="1.838168855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:19:36.836992275 +0000 UTC m=+3241.740674390" watchObservedRunningTime="2026-11-02 02:19:36.838168855 +0000 UTC m=+3241.741850969"
Nov 02 02:19:37.728707 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:37.728664    5639 generic.go:356] "Generic (PLEG): container finished" podID="08ef4ca8-74b3-4d03-b24d-2d421d42edf4" containerID="99b29efa271caadfc9b614c5e8ed8b60ad3a5c25490b5b57d5c5b7a736180e87" exitCode=0
Nov 02 02:19:37.729200 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:37.728752    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lqjpk" event={"ID":"08ef4ca8-74b3-4d03-b24d-2d421d42edf4","Type":"ContainerDied","Data":"99b29efa271caadfc9b614c5e8ed8b60ad3a5c25490b5b57d5c5b7a736180e87"}
Nov 02 02:19:38.885683 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:38.885656    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lqjpk"
Nov 02 02:19:38.965418 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:38.965386    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ef4ca8-74b3-4d03-b24d-2d421d42edf4-kube-api-access-x2jb8\" (UniqueName: \"kubernetes.io/projected/08ef4ca8-74b3-4d03-b24d-2d421d42edf4-kube-api-access-x2jb8\") pod \"08ef4ca8-74b3-4d03-b24d-2d421d42edf4\" (UID: \"08ef4ca8-74b3-4d03-b24d-2d421d42edf4\") "
Nov 02 02:19:38.965626 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:38.965442    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08ef4ca8-74b3-4d03-b24d-2d421d42edf4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ef4ca8-74b3-4d03-b24d-2d421d42edf4-maas-api-openshift-ca\") pod \"08ef4ca8-74b3-4d03-b24d-2d421d42edf4\" (UID: \"08ef4ca8-74b3-4d03-b24d-2d421d42edf4\") "
Nov 02 02:19:38.965626 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:38.965500    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08ef4ca8-74b3-4d03-b24d-2d421d42edf4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ef4ca8-74b3-4d03-b24d-2d421d42edf4-maas-api-xks-ca\") pod \"08ef4ca8-74b3-4d03-b24d-2d421d42edf4\" (UID: \"08ef4ca8-74b3-4d03-b24d-2d421d42edf4\") "
Nov 02 02:19:38.965854 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:38.965831    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ef4ca8-74b3-4d03-b24d-2d421d42edf4-maas-api-openshift-ca" pod "08ef4ca8-74b3-4d03-b24d-2d421d42edf4" (UID: "08ef4ca8-74b3-4d03-b24d-2d421d42edf4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:19:38.967663 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:38.967637    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ef4ca8-74b3-4d03-b24d-2d421d42edf4-maas-api-xks-ca" pod "08ef4ca8-74b3-4d03-b24d-2d421d42edf4" (UID: "08ef4ca8-74b3-4d03-b24d-2d421d42edf4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:19:38.967799 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:38.967776    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ef4ca8-74b3-4d03-b24d-2d421d42edf4-kube-api-access-x2jb8" pod "08ef4ca8-74b3-4d03-b24d-2d421d42edf4" (UID: "08ef4ca8-74b3-4d03-b24d-2d421d42edf4"). InnerVolumeSpecName "kube-api-access-x2jb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:19:39.066615 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:39.066528    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2jb8\" (UniqueName: \"kubernetes.io/projected/08ef4ca8-74b3-4d03-b24d-2d421d42edf4-kube-api-access-x2jb8\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:19:39.066615 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:39.066558    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ef4ca8-74b3-4d03-b24d-2d421d42edf4-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:19:39.066615 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:39.066570    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ef4ca8-74b3-4d03-b24d-2d421d42edf4-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:19:39.338572 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:39.338545    5639 scope.go:122] "RemoveContainer" containerID="46b85f8b26bb4eb939a030fc1bf3048c47bf0a15b86d374d7721df0ebe14e144"
Nov 02 02:19:39.347757 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:39.347736    5639 scope.go:122] "RemoveContainer" containerID="4e363bde5862cf6f5bddda79a1cba82802ed69a76b8d0b78dd92731519bdfd69"
Nov 02 02:19:39.740237 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:39.740153    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lqjpk"
Nov 02 02:19:39.740237 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:39.740197    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lqjpk" event={"ID":"08ef4ca8-74b3-4d03-b24d-2d421d42edf4","Type":"ContainerDied","Data":"06c3877df214529abedcb57d011cffc97290485f482db202862e4f38adf40927"}
Nov 02 02:19:39.740457 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:39.740242    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06c3877df214529abedcb57d011cffc97290485f482db202862e4f38adf40927"
Nov 02 02:19:49.983631 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:49.983549    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:19:57.420235 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:19:57.420194    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:19:59.240925 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:59.240880    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fcbfq"]
Nov 02 02:19:59.241593 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:59.241343    5639 cpu_manager.go:414] "containerMap: removing container" podUID="08ef4ca8-74b3-4d03-b24d-2d421d42edf4" containerName="revoke-keys"
Nov 02 02:19:59.241593 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:59.241356    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ef4ca8-74b3-4d03-b24d-2d421d42edf4" containerName="revoke-keys"
Nov 02 02:19:59.241593 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:59.241473    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ef4ca8-74b3-4d03-b24d-2d421d42edf4" containerName="revoke-keys"
Nov 02 02:19:59.245885 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:59.245864    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fcbfq"
Nov 02 02:19:59.302081 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:59.302049    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fcbfq"]
Nov 02 02:19:59.349077 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:59.349044    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:19:59.349077 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:59.349077    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:19:59.349359 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:59.349247    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:19:59.349359 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:59.349332    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:19:59.416807 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:59.416777    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8ec2711-d219-47b2-90b6-13310e016f3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fcbfq\" (UID: \"e8ec2711-d219-47b2-90b6-13310e016f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fcbfq"
Nov 02 02:19:59.416807 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:59.416807    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w67jl\" (UniqueName: \"kubernetes.io/projected/e8ec2711-d219-47b2-90b6-13310e016f3f-kube-api-access-w67jl\") pod \"maas-api-revoke-sub-0b7e8edb-fcbfq\" (UID: \"e8ec2711-d219-47b2-90b6-13310e016f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fcbfq"
Nov 02 02:19:59.417024 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:59.416933    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8ec2711-d219-47b2-90b6-13310e016f3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fcbfq\" (UID: \"e8ec2711-d219-47b2-90b6-13310e016f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fcbfq"
Nov 02 02:19:59.518387 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:59.518270    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8ec2711-d219-47b2-90b6-13310e016f3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fcbfq\" (UID: \"e8ec2711-d219-47b2-90b6-13310e016f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fcbfq"
Nov 02 02:19:59.518387 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:59.518330    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w67jl\" (UniqueName: \"kubernetes.io/projected/e8ec2711-d219-47b2-90b6-13310e016f3f-kube-api-access-w67jl\") pod \"maas-api-revoke-sub-0b7e8edb-fcbfq\" (UID: \"e8ec2711-d219-47b2-90b6-13310e016f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fcbfq"
Nov 02 02:19:59.518607 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:59.518413    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8ec2711-d219-47b2-90b6-13310e016f3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fcbfq\" (UID: \"e8ec2711-d219-47b2-90b6-13310e016f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fcbfq"
Nov 02 02:19:59.518907 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:59.518886    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8ec2711-d219-47b2-90b6-13310e016f3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fcbfq\" (UID: \"e8ec2711-d219-47b2-90b6-13310e016f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fcbfq"
Nov 02 02:19:59.520849 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:59.520829    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8ec2711-d219-47b2-90b6-13310e016f3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fcbfq\" (UID: \"e8ec2711-d219-47b2-90b6-13310e016f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fcbfq"
Nov 02 02:19:59.579585 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:59.579551    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w67jl\" (UniqueName: \"kubernetes.io/projected/e8ec2711-d219-47b2-90b6-13310e016f3f-kube-api-access-w67jl\") pod \"maas-api-revoke-sub-0b7e8edb-fcbfq\" (UID: \"e8ec2711-d219-47b2-90b6-13310e016f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fcbfq"
Nov 02 02:19:59.858694 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:19:59.858664    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fcbfq"
Nov 02 02:20:00.235227 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:00.235200    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fcbfq"]
Nov 02 02:20:00.237421 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:20:00.237393    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ec2711_d219_47b2_90b6_13310e016f3f.slice/crio-b2aaa0febfe71b584f0eb748c5497a367210fca2e1d52649cb3f1635a3bbb725 WatchSource:0}: Error finding container b2aaa0febfe71b584f0eb748c5497a367210fca2e1d52649cb3f1635a3bbb725: Status 404 returned error can't find the container with id b2aaa0febfe71b584f0eb748c5497a367210fca2e1d52649cb3f1635a3bbb725
Nov 02 02:20:00.897508 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:00.897470    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fcbfq" event={"ID":"e8ec2711-d219-47b2-90b6-13310e016f3f","Type":"ContainerStarted","Data":"714557a30e85dd8550600552be5dfb59bdd458434af479ea73c9efdbf089e7bf"}
Nov 02 02:20:00.898025 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:00.897528    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fcbfq" event={"ID":"e8ec2711-d219-47b2-90b6-13310e016f3f","Type":"ContainerStarted","Data":"b2aaa0febfe71b584f0eb748c5497a367210fca2e1d52649cb3f1635a3bbb725"}
Nov 02 02:20:01.013847 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:01.013796    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fcbfq" podStartSLOduration=2.013781466 podStartE2EDuration="2.013781466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:20:01.01194585 +0000 UTC m=+3265.915627965" watchObservedRunningTime="2026-11-02 02:20:01.013781466 +0000 UTC m=+3265.917463579"
Nov 02 02:20:01.904030 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:01.903993    5639 generic.go:356] "Generic (PLEG): container finished" podID="e8ec2711-d219-47b2-90b6-13310e016f3f" containerID="714557a30e85dd8550600552be5dfb59bdd458434af479ea73c9efdbf089e7bf" exitCode=0
Nov 02 02:20:01.904030 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:01.904033    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fcbfq" event={"ID":"e8ec2711-d219-47b2-90b6-13310e016f3f","Type":"ContainerDied","Data":"714557a30e85dd8550600552be5dfb59bdd458434af479ea73c9efdbf089e7bf"}
Nov 02 02:20:03.049688 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:03.049656    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fcbfq"
Nov 02 02:20:03.050120 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:03.050100    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8ec2711-d219-47b2-90b6-13310e016f3f-kube-api-access-w67jl\" (UniqueName: \"kubernetes.io/projected/e8ec2711-d219-47b2-90b6-13310e016f3f-kube-api-access-w67jl\") pod \"e8ec2711-d219-47b2-90b6-13310e016f3f\" (UID: \"e8ec2711-d219-47b2-90b6-13310e016f3f\") "
Nov 02 02:20:03.050191 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:03.050133    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8ec2711-d219-47b2-90b6-13310e016f3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8ec2711-d219-47b2-90b6-13310e016f3f-maas-api-openshift-ca\") pod \"e8ec2711-d219-47b2-90b6-13310e016f3f\" (UID: \"e8ec2711-d219-47b2-90b6-13310e016f3f\") "
Nov 02 02:20:03.050255 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:03.050240    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8ec2711-d219-47b2-90b6-13310e016f3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8ec2711-d219-47b2-90b6-13310e016f3f-maas-api-xks-ca\") pod \"e8ec2711-d219-47b2-90b6-13310e016f3f\" (UID: \"e8ec2711-d219-47b2-90b6-13310e016f3f\") "
Nov 02 02:20:03.050544 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:03.050521    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ec2711-d219-47b2-90b6-13310e016f3f-maas-api-openshift-ca" pod "e8ec2711-d219-47b2-90b6-13310e016f3f" (UID: "e8ec2711-d219-47b2-90b6-13310e016f3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:20:03.052430 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:03.052408    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8ec2711-d219-47b2-90b6-13310e016f3f-maas-api-xks-ca" pod "e8ec2711-d219-47b2-90b6-13310e016f3f" (UID: "e8ec2711-d219-47b2-90b6-13310e016f3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:20:03.052589 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:03.052568    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ec2711-d219-47b2-90b6-13310e016f3f-kube-api-access-w67jl" pod "e8ec2711-d219-47b2-90b6-13310e016f3f" (UID: "e8ec2711-d219-47b2-90b6-13310e016f3f"). InnerVolumeSpecName "kube-api-access-w67jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:20:03.151580 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:03.151541    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8ec2711-d219-47b2-90b6-13310e016f3f-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:20:03.151743 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:03.151608    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w67jl\" (UniqueName: \"kubernetes.io/projected/e8ec2711-d219-47b2-90b6-13310e016f3f-kube-api-access-w67jl\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:20:03.151743 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:03.151627    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8ec2711-d219-47b2-90b6-13310e016f3f-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:20:03.915734 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:03.915708    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fcbfq"
Nov 02 02:20:03.915926 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:03.915700    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fcbfq" event={"ID":"e8ec2711-d219-47b2-90b6-13310e016f3f","Type":"ContainerDied","Data":"b2aaa0febfe71b584f0eb748c5497a367210fca2e1d52649cb3f1635a3bbb725"}
Nov 02 02:20:03.915926 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:03.915833    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2aaa0febfe71b584f0eb748c5497a367210fca2e1d52649cb3f1635a3bbb725"
Nov 02 02:20:10.598773 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:10.598733    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt69g"]
Nov 02 02:20:10.599244 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:10.599229    5639 cpu_manager.go:414] "containerMap: removing container" podUID="e8ec2711-d219-47b2-90b6-13310e016f3f" containerName="revoke-keys"
Nov 02 02:20:10.599316 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:10.599245    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8ec2711-d219-47b2-90b6-13310e016f3f" containerName="revoke-keys"
Nov 02 02:20:10.599383 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:10.599372    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8ec2711-d219-47b2-90b6-13310e016f3f" containerName="revoke-keys"
Nov 02 02:20:10.602779 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:10.602760    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt69g"
Nov 02 02:20:10.629494 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:10.629450    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7xcn\" (UniqueName: \"kubernetes.io/projected/d7382348-a0d5-47ff-a696-a26486f3b08c-kube-api-access-s7xcn\") pod \"maas-api-revoke-sub-3d4156f1-lt69g\" (UID: \"d7382348-a0d5-47ff-a696-a26486f3b08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt69g"
Nov 02 02:20:10.629662 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:10.629504    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7382348-a0d5-47ff-a696-a26486f3b08c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lt69g\" (UID: \"d7382348-a0d5-47ff-a696-a26486f3b08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt69g"
Nov 02 02:20:10.629713 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:10.629676    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7382348-a0d5-47ff-a696-a26486f3b08c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lt69g\" (UID: \"d7382348-a0d5-47ff-a696-a26486f3b08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt69g"
Nov 02 02:20:10.696673 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:10.696626    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:20:10.707292 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:10.707240    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:20:10.707495 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:10.707320    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:20:10.707495 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:10.707416    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:20:10.709652 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:10.709634    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt69g"]
Nov 02 02:20:10.730749 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:10.730713    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7xcn\" (UniqueName: \"kubernetes.io/projected/d7382348-a0d5-47ff-a696-a26486f3b08c-kube-api-access-s7xcn\") pod \"maas-api-revoke-sub-3d4156f1-lt69g\" (UID: \"d7382348-a0d5-47ff-a696-a26486f3b08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt69g"
Nov 02 02:20:10.730749 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:10.730752    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7382348-a0d5-47ff-a696-a26486f3b08c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lt69g\" (UID: \"d7382348-a0d5-47ff-a696-a26486f3b08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt69g"
Nov 02 02:20:10.731044 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:10.730841    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7382348-a0d5-47ff-a696-a26486f3b08c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lt69g\" (UID: \"d7382348-a0d5-47ff-a696-a26486f3b08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt69g"
Nov 02 02:20:10.731482 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:10.731458    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7382348-a0d5-47ff-a696-a26486f3b08c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lt69g\" (UID: \"d7382348-a0d5-47ff-a696-a26486f3b08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt69g"
Nov 02 02:20:10.733569 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:10.733544    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7382348-a0d5-47ff-a696-a26486f3b08c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lt69g\" (UID: \"d7382348-a0d5-47ff-a696-a26486f3b08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt69g"
Nov 02 02:20:10.792602 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:10.792571    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7xcn\" (UniqueName: \"kubernetes.io/projected/d7382348-a0d5-47ff-a696-a26486f3b08c-kube-api-access-s7xcn\") pod \"maas-api-revoke-sub-3d4156f1-lt69g\" (UID: \"d7382348-a0d5-47ff-a696-a26486f3b08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt69g"
Nov 02 02:20:10.916973 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:10.916877    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt69g"
Nov 02 02:20:11.294268 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:11.294239    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt69g"]
Nov 02 02:20:11.296392 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:20:11.296359    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7382348_a0d5_47ff_a696_a26486f3b08c.slice/crio-43101b2c51d64b7310013b204ed3b0650e1e52e93779f96e565d379c66d2f590 WatchSource:0}: Error finding container 43101b2c51d64b7310013b204ed3b0650e1e52e93779f96e565d379c66d2f590: Status 404 returned error can't find the container with id 43101b2c51d64b7310013b204ed3b0650e1e52e93779f96e565d379c66d2f590
Nov 02 02:20:11.960994 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:11.960958    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt69g" event={"ID":"d7382348-a0d5-47ff-a696-a26486f3b08c","Type":"ContainerStarted","Data":"c569fc17da2f2cae601872ee5b0709a9870f7bec9b243098582ed9dc1dfc9431"}
Nov 02 02:20:11.961460 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:11.961018    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt69g" event={"ID":"d7382348-a0d5-47ff-a696-a26486f3b08c","Type":"ContainerStarted","Data":"43101b2c51d64b7310013b204ed3b0650e1e52e93779f96e565d379c66d2f590"}
Nov 02 02:20:12.078394 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:12.078322    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt69g" podStartSLOduration=2.078308354 podStartE2EDuration="2.078308354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:20:12.076537548 +0000 UTC m=+3276.980219686" watchObservedRunningTime="2026-11-02 02:20:12.078308354 +0000 UTC m=+3276.981990468"
Nov 02 02:20:12.104756 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:12.104719    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:20:13.016647 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:13.016606    5639 generic.go:356] "Generic (PLEG): container finished" podID="d7382348-a0d5-47ff-a696-a26486f3b08c" containerID="c569fc17da2f2cae601872ee5b0709a9870f7bec9b243098582ed9dc1dfc9431" exitCode=0
Nov 02 02:20:13.017355 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:13.017326    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt69g" event={"ID":"d7382348-a0d5-47ff-a696-a26486f3b08c","Type":"ContainerDied","Data":"c569fc17da2f2cae601872ee5b0709a9870f7bec9b243098582ed9dc1dfc9431"}
Nov 02 02:20:14.511641 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:14.511612    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt69g"
Nov 02 02:20:14.585720 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:14.585685    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7382348-a0d5-47ff-a696-a26486f3b08c-kube-api-access-s7xcn\" (UniqueName: \"kubernetes.io/projected/d7382348-a0d5-47ff-a696-a26486f3b08c-kube-api-access-s7xcn\") pod \"d7382348-a0d5-47ff-a696-a26486f3b08c\" (UID: \"d7382348-a0d5-47ff-a696-a26486f3b08c\") "
Nov 02 02:20:14.585936 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:14.585760    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7382348-a0d5-47ff-a696-a26486f3b08c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7382348-a0d5-47ff-a696-a26486f3b08c-maas-api-openshift-ca\") pod \"d7382348-a0d5-47ff-a696-a26486f3b08c\" (UID: \"d7382348-a0d5-47ff-a696-a26486f3b08c\") "
Nov 02 02:20:14.585936 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:14.585821    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7382348-a0d5-47ff-a696-a26486f3b08c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7382348-a0d5-47ff-a696-a26486f3b08c-maas-api-xks-ca\") pod \"d7382348-a0d5-47ff-a696-a26486f3b08c\" (UID: \"d7382348-a0d5-47ff-a696-a26486f3b08c\") "
Nov 02 02:20:14.586164 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:14.586135    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7382348-a0d5-47ff-a696-a26486f3b08c-maas-api-openshift-ca" pod "d7382348-a0d5-47ff-a696-a26486f3b08c" (UID: "d7382348-a0d5-47ff-a696-a26486f3b08c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:20:14.588152 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:14.588122    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7382348-a0d5-47ff-a696-a26486f3b08c-maas-api-xks-ca" pod "d7382348-a0d5-47ff-a696-a26486f3b08c" (UID: "d7382348-a0d5-47ff-a696-a26486f3b08c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:20:14.588315 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:14.588157    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7382348-a0d5-47ff-a696-a26486f3b08c-kube-api-access-s7xcn" pod "d7382348-a0d5-47ff-a696-a26486f3b08c" (UID: "d7382348-a0d5-47ff-a696-a26486f3b08c"). InnerVolumeSpecName "kube-api-access-s7xcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:20:14.687612 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:14.687514    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7xcn\" (UniqueName: \"kubernetes.io/projected/d7382348-a0d5-47ff-a696-a26486f3b08c-kube-api-access-s7xcn\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:20:14.687612 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:14.687549    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7382348-a0d5-47ff-a696-a26486f3b08c-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:20:14.687612 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:14.687561    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7382348-a0d5-47ff-a696-a26486f3b08c-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:20:15.058259 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:15.058173    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt69g"
Nov 02 02:20:15.058259 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:15.058203    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt69g" event={"ID":"d7382348-a0d5-47ff-a696-a26486f3b08c","Type":"ContainerDied","Data":"43101b2c51d64b7310013b204ed3b0650e1e52e93779f96e565d379c66d2f590"}
Nov 02 02:20:15.058259 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:15.058258    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43101b2c51d64b7310013b204ed3b0650e1e52e93779f96e565d379c66d2f590"
Nov 02 02:20:25.049389 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:25.049348    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cmhhj"]
Nov 02 02:20:25.099427 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:25.099398    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cmhhj"]
Nov 02 02:20:25.217339 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:25.217301    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17febcbd-d3c4-4b47-9877-948fb5607524" path="/var/lib/kubelet/pods/17febcbd-d3c4-4b47-9877-948fb5607524/volumes"
Nov 02 02:20:25.221339 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:25.221307    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:20:27.481310 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:20:27.480689    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:20:34.102307 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:34.101397    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:20:34.102307 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:34.101489    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7" podUID="70ec0d83-9e6c-4e20-82b6-2f09909fc85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:20:35.906730 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:35.906694    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:20:39.372166 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:39.372128    5639 scope.go:122] "RemoveContainer" containerID="260841523bd00af853c5375ae13951c23b538d7994402c050b38d05b170c0ac8"
Nov 02 02:20:39.381592 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:39.381574    5639 scope.go:122] "RemoveContainer" containerID="dd21238e36806988c9523009c03b951566a4c4b45ca916307c7a6865e8558db7"
Nov 02 02:20:39.393532 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:39.393504    5639 scope.go:122] "RemoveContainer" containerID="fb75dded064da9124e8285d0bdc3dac25437109d1e18ae17de7a2287dd875234"
Nov 02 02:20:44.364386 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:44.364343    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt69g"]
Nov 02 02:20:44.413560 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:44.413529    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt69g"]
Nov 02 02:20:44.486563 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:44.486525    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j45c6"]
Nov 02 02:20:44.487004 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:44.486990    5639 cpu_manager.go:414] "containerMap: removing container" podUID="d7382348-a0d5-47ff-a696-a26486f3b08c" containerName="revoke-keys"
Nov 02 02:20:44.487076 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:44.487005    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7382348-a0d5-47ff-a696-a26486f3b08c" containerName="revoke-keys"
Nov 02 02:20:44.487076 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:44.487075    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7382348-a0d5-47ff-a696-a26486f3b08c" containerName="revoke-keys"
Nov 02 02:20:44.491717 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:44.491694    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j45c6"
Nov 02 02:20:44.546159 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:44.546122    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:20:44.546482 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:44.546243    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:20:44.546482 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:44.546243    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:20:44.546482 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:44.546358    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:20:44.549216 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:44.549188    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j45c6"]
Nov 02 02:20:44.596805 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:44.596770    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gdfg\" (UniqueName: \"kubernetes.io/projected/51579b3c-5e8a-49cd-b8ab-01484d7eecbb-kube-api-access-6gdfg\") pod \"maas-api-revoke-sub-3d4156f1-j45c6\" (UID: \"51579b3c-5e8a-49cd-b8ab-01484d7eecbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j45c6"
Nov 02 02:20:44.597025 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:44.596857    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51579b3c-5e8a-49cd-b8ab-01484d7eecbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-j45c6\" (UID: \"51579b3c-5e8a-49cd-b8ab-01484d7eecbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j45c6"
Nov 02 02:20:44.597025 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:44.596964    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51579b3c-5e8a-49cd-b8ab-01484d7eecbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-j45c6\" (UID: \"51579b3c-5e8a-49cd-b8ab-01484d7eecbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j45c6"
Nov 02 02:20:44.698081 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:44.697989    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51579b3c-5e8a-49cd-b8ab-01484d7eecbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-j45c6\" (UID: \"51579b3c-5e8a-49cd-b8ab-01484d7eecbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j45c6"
Nov 02 02:20:44.698081 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:44.698040    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51579b3c-5e8a-49cd-b8ab-01484d7eecbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-j45c6\" (UID: \"51579b3c-5e8a-49cd-b8ab-01484d7eecbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j45c6"
Nov 02 02:20:44.698383 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:44.698092    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gdfg\" (UniqueName: \"kubernetes.io/projected/51579b3c-5e8a-49cd-b8ab-01484d7eecbb-kube-api-access-6gdfg\") pod \"maas-api-revoke-sub-3d4156f1-j45c6\" (UID: \"51579b3c-5e8a-49cd-b8ab-01484d7eecbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j45c6"
Nov 02 02:20:44.698804 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:44.698774    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51579b3c-5e8a-49cd-b8ab-01484d7eecbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-j45c6\" (UID: \"51579b3c-5e8a-49cd-b8ab-01484d7eecbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j45c6"
Nov 02 02:20:44.700696 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:44.700663    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51579b3c-5e8a-49cd-b8ab-01484d7eecbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-j45c6\" (UID: \"51579b3c-5e8a-49cd-b8ab-01484d7eecbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j45c6"
Nov 02 02:20:44.759694 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:44.759658    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gdfg\" (UniqueName: \"kubernetes.io/projected/51579b3c-5e8a-49cd-b8ab-01484d7eecbb-kube-api-access-6gdfg\") pod \"maas-api-revoke-sub-3d4156f1-j45c6\" (UID: \"51579b3c-5e8a-49cd-b8ab-01484d7eecbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j45c6"
Nov 02 02:20:44.805068 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:44.805029    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j45c6"
Nov 02 02:20:45.181748 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:45.181721    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j45c6"]
Nov 02 02:20:45.184183 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:20:45.184154    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51579b3c_5e8a_49cd_b8ab_01484d7eecbb.slice/crio-0e62969932d537d72d6295513e95133ffe901d9b8284c64514e1880cbe144e44 WatchSource:0}: Error finding container 0e62969932d537d72d6295513e95133ffe901d9b8284c64514e1880cbe144e44: Status 404 returned error can't find the container with id 0e62969932d537d72d6295513e95133ffe901d9b8284c64514e1880cbe144e44
Nov 02 02:20:45.218178 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:45.218148    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7382348-a0d5-47ff-a696-a26486f3b08c" path="/var/lib/kubelet/pods/d7382348-a0d5-47ff-a696-a26486f3b08c/volumes"
Nov 02 02:20:45.270370 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:45.270325    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j45c6" event={"ID":"51579b3c-5e8a-49cd-b8ab-01484d7eecbb","Type":"ContainerStarted","Data":"0e62969932d537d72d6295513e95133ffe901d9b8284c64514e1880cbe144e44"}
Nov 02 02:20:46.277406 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:46.277304    5639 generic.go:356] "Generic (PLEG): container finished" podID="51579b3c-5e8a-49cd-b8ab-01484d7eecbb" containerID="d4b100017bcbc4c6479e3a26705192c207b9ded0cd40d30fbec8c7c41bf18241" exitCode=0
Nov 02 02:20:46.277798 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:46.277404    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j45c6" event={"ID":"51579b3c-5e8a-49cd-b8ab-01484d7eecbb","Type":"ContainerDied","Data":"d4b100017bcbc4c6479e3a26705192c207b9ded0cd40d30fbec8c7c41bf18241"}
Nov 02 02:20:47.418955 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:47.418928    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j45c6"
Nov 02 02:20:47.526380 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:47.526338    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51579b3c-5e8a-49cd-b8ab-01484d7eecbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51579b3c-5e8a-49cd-b8ab-01484d7eecbb-maas-api-xks-ca\") pod \"51579b3c-5e8a-49cd-b8ab-01484d7eecbb\" (UID: \"51579b3c-5e8a-49cd-b8ab-01484d7eecbb\") "
Nov 02 02:20:47.526553 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:47.526431    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51579b3c-5e8a-49cd-b8ab-01484d7eecbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51579b3c-5e8a-49cd-b8ab-01484d7eecbb-maas-api-openshift-ca\") pod \"51579b3c-5e8a-49cd-b8ab-01484d7eecbb\" (UID: \"51579b3c-5e8a-49cd-b8ab-01484d7eecbb\") "
Nov 02 02:20:47.526553 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:47.526528    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51579b3c-5e8a-49cd-b8ab-01484d7eecbb-kube-api-access-6gdfg\" (UniqueName: \"kubernetes.io/projected/51579b3c-5e8a-49cd-b8ab-01484d7eecbb-kube-api-access-6gdfg\") pod \"51579b3c-5e8a-49cd-b8ab-01484d7eecbb\" (UID: \"51579b3c-5e8a-49cd-b8ab-01484d7eecbb\") "
Nov 02 02:20:47.526813 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:47.526790    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51579b3c-5e8a-49cd-b8ab-01484d7eecbb-maas-api-openshift-ca" pod "51579b3c-5e8a-49cd-b8ab-01484d7eecbb" (UID: "51579b3c-5e8a-49cd-b8ab-01484d7eecbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:20:47.526939 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:47.526926    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51579b3c-5e8a-49cd-b8ab-01484d7eecbb-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:20:47.528790 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:47.528767    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51579b3c-5e8a-49cd-b8ab-01484d7eecbb-maas-api-xks-ca" pod "51579b3c-5e8a-49cd-b8ab-01484d7eecbb" (UID: "51579b3c-5e8a-49cd-b8ab-01484d7eecbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:20:47.528908 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:47.528887    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51579b3c-5e8a-49cd-b8ab-01484d7eecbb-kube-api-access-6gdfg" pod "51579b3c-5e8a-49cd-b8ab-01484d7eecbb" (UID: "51579b3c-5e8a-49cd-b8ab-01484d7eecbb"). InnerVolumeSpecName "kube-api-access-6gdfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:20:47.627619 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:47.627577    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gdfg\" (UniqueName: \"kubernetes.io/projected/51579b3c-5e8a-49cd-b8ab-01484d7eecbb-kube-api-access-6gdfg\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:20:47.627619 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:47.627609    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51579b3c-5e8a-49cd-b8ab-01484d7eecbb-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:20:48.288730 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:48.288691    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j45c6"
Nov 02 02:20:48.288913 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:48.288719    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j45c6" event={"ID":"51579b3c-5e8a-49cd-b8ab-01484d7eecbb","Type":"ContainerDied","Data":"0e62969932d537d72d6295513e95133ffe901d9b8284c64514e1880cbe144e44"}
Nov 02 02:20:48.288913 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:48.288813    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e62969932d537d72d6295513e95133ffe901d9b8284c64514e1880cbe144e44"
Nov 02 02:20:57.421398 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:20:57.421353    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:20:59.097489 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:20:59.097452    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:21:09.282323 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:09.282272    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:21:24.703759 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:24.703677    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:21:27.472676 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:21:27.472606    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:21:35.255144 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:35.255101    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-j8msj"]
Nov 02 02:21:35.255678 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:35.255660    5639 cpu_manager.go:414] "containerMap: removing container" podUID="51579b3c-5e8a-49cd-b8ab-01484d7eecbb" containerName="revoke-keys"
Nov 02 02:21:35.255678 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:35.255675    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51579b3c-5e8a-49cd-b8ab-01484d7eecbb" containerName="revoke-keys"
Nov 02 02:21:35.255825 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:35.255772    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="51579b3c-5e8a-49cd-b8ab-01484d7eecbb" containerName="revoke-keys"
Nov 02 02:21:35.259173 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:35.259152    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-j8msj"
Nov 02 02:21:35.316975 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:35.316939    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-j8msj"]
Nov 02 02:21:35.362949 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:35.362910    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:21:35.363135 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:35.362975    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:21:35.363135 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:35.363007    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:21:35.363135 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:35.363027    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:21:35.441163 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:35.441119    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea3a2c94-a15c-4214-8485-04316281816f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-j8msj\" (UID: \"ea3a2c94-a15c-4214-8485-04316281816f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-j8msj"
Nov 02 02:21:35.441413 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:35.441314    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z8pz\" (UniqueName: \"kubernetes.io/projected/ea3a2c94-a15c-4214-8485-04316281816f-kube-api-access-6z8pz\") pod \"maas-api-revoke-sub-e2fc9c5b-j8msj\" (UID: \"ea3a2c94-a15c-4214-8485-04316281816f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-j8msj"
Nov 02 02:21:35.441413 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:35.441366    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea3a2c94-a15c-4214-8485-04316281816f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-j8msj\" (UID: \"ea3a2c94-a15c-4214-8485-04316281816f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-j8msj"
Nov 02 02:21:35.542747 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:35.542650    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea3a2c94-a15c-4214-8485-04316281816f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-j8msj\" (UID: \"ea3a2c94-a15c-4214-8485-04316281816f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-j8msj"
Nov 02 02:21:35.542914 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:35.542754    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z8pz\" (UniqueName: \"kubernetes.io/projected/ea3a2c94-a15c-4214-8485-04316281816f-kube-api-access-6z8pz\") pod \"maas-api-revoke-sub-e2fc9c5b-j8msj\" (UID: \"ea3a2c94-a15c-4214-8485-04316281816f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-j8msj"
Nov 02 02:21:35.542914 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:35.542781    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea3a2c94-a15c-4214-8485-04316281816f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-j8msj\" (UID: \"ea3a2c94-a15c-4214-8485-04316281816f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-j8msj"
Nov 02 02:21:35.543491 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:35.543456    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea3a2c94-a15c-4214-8485-04316281816f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-j8msj\" (UID: \"ea3a2c94-a15c-4214-8485-04316281816f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-j8msj"
Nov 02 02:21:35.545469 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:35.545440    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea3a2c94-a15c-4214-8485-04316281816f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-j8msj\" (UID: \"ea3a2c94-a15c-4214-8485-04316281816f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-j8msj"
Nov 02 02:21:35.604357 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:35.604320    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z8pz\" (UniqueName: \"kubernetes.io/projected/ea3a2c94-a15c-4214-8485-04316281816f-kube-api-access-6z8pz\") pod \"maas-api-revoke-sub-e2fc9c5b-j8msj\" (UID: \"ea3a2c94-a15c-4214-8485-04316281816f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-j8msj"
Nov 02 02:21:35.873292 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:35.873251    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-j8msj"
Nov 02 02:21:36.063319 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:36.063268    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rvvf9"]
Nov 02 02:21:36.065777 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:36.065750    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rvvf9"]
Nov 02 02:21:36.249798 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:36.249767    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-j8msj"]
Nov 02 02:21:36.252007 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:21:36.251966    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3a2c94_a15c_4214_8485_04316281816f.slice/crio-ec2ce40dd5672a0e29e4dbf62a0a5a82fa36268cf2caacce6ed4ac42040784d1 WatchSource:0}: Error finding container ec2ce40dd5672a0e29e4dbf62a0a5a82fa36268cf2caacce6ed4ac42040784d1: Status 404 returned error can't find the container with id ec2ce40dd5672a0e29e4dbf62a0a5a82fa36268cf2caacce6ed4ac42040784d1
Nov 02 02:21:36.587684 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:36.587649    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-j8msj" event={"ID":"ea3a2c94-a15c-4214-8485-04316281816f","Type":"ContainerStarted","Data":"1be628a4c5dc3f4450a980c4104f39253708dd4545e5c2ca36f13f8b280b36c4"}
Nov 02 02:21:36.588063 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:36.587712    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-j8msj" event={"ID":"ea3a2c94-a15c-4214-8485-04316281816f","Type":"ContainerStarted","Data":"ec2ce40dd5672a0e29e4dbf62a0a5a82fa36268cf2caacce6ed4ac42040784d1"}
Nov 02 02:21:36.712186 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:36.712127    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-j8msj" podStartSLOduration=1.712113529 podStartE2EDuration="1.712113529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:21:36.710308903 +0000 UTC m=+3361.613991012" watchObservedRunningTime="2026-11-02 02:21:36.712113529 +0000 UTC m=+3361.615795640"
Nov 02 02:21:37.218198 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:37.218166    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99d4b035-486d-4d58-a5e3-2f4eafab38ca" path="/var/lib/kubelet/pods/99d4b035-486d-4d58-a5e3-2f4eafab38ca/volumes"
Nov 02 02:21:37.599493 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:37.599458    5639 generic.go:356] "Generic (PLEG): container finished" podID="ea3a2c94-a15c-4214-8485-04316281816f" containerID="1be628a4c5dc3f4450a980c4104f39253708dd4545e5c2ca36f13f8b280b36c4" exitCode=0
Nov 02 02:21:37.599993 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:37.599558    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-j8msj" event={"ID":"ea3a2c94-a15c-4214-8485-04316281816f","Type":"ContainerDied","Data":"1be628a4c5dc3f4450a980c4104f39253708dd4545e5c2ca36f13f8b280b36c4"}
Nov 02 02:21:38.728832 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:38.728805    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-j8msj"
Nov 02 02:21:38.773013 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:38.772983    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea3a2c94-a15c-4214-8485-04316281816f-kube-api-access-6z8pz\" (UniqueName: \"kubernetes.io/projected/ea3a2c94-a15c-4214-8485-04316281816f-kube-api-access-6z8pz\") pod \"ea3a2c94-a15c-4214-8485-04316281816f\" (UID: \"ea3a2c94-a15c-4214-8485-04316281816f\") "
Nov 02 02:21:38.773196 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:38.773022    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea3a2c94-a15c-4214-8485-04316281816f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea3a2c94-a15c-4214-8485-04316281816f-maas-api-openshift-ca\") pod \"ea3a2c94-a15c-4214-8485-04316281816f\" (UID: \"ea3a2c94-a15c-4214-8485-04316281816f\") "
Nov 02 02:21:38.773196 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:38.773099    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea3a2c94-a15c-4214-8485-04316281816f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea3a2c94-a15c-4214-8485-04316281816f-maas-api-xks-ca\") pod \"ea3a2c94-a15c-4214-8485-04316281816f\" (UID: \"ea3a2c94-a15c-4214-8485-04316281816f\") "
Nov 02 02:21:38.773487 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:38.773462    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea3a2c94-a15c-4214-8485-04316281816f-maas-api-openshift-ca" pod "ea3a2c94-a15c-4214-8485-04316281816f" (UID: "ea3a2c94-a15c-4214-8485-04316281816f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:21:38.775297 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:38.775262    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea3a2c94-a15c-4214-8485-04316281816f-maas-api-xks-ca" pod "ea3a2c94-a15c-4214-8485-04316281816f" (UID: "ea3a2c94-a15c-4214-8485-04316281816f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:21:38.775389 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:38.775373    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3a2c94-a15c-4214-8485-04316281816f-kube-api-access-6z8pz" pod "ea3a2c94-a15c-4214-8485-04316281816f" (UID: "ea3a2c94-a15c-4214-8485-04316281816f"). InnerVolumeSpecName "kube-api-access-6z8pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:21:38.874695 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:38.874604    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z8pz\" (UniqueName: \"kubernetes.io/projected/ea3a2c94-a15c-4214-8485-04316281816f-kube-api-access-6z8pz\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:21:38.874695 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:38.874633    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea3a2c94-a15c-4214-8485-04316281816f-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:21:38.874695 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:38.874647    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea3a2c94-a15c-4214-8485-04316281816f-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:21:39.429191 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:39.429159    5639 scope.go:122] "RemoveContainer" containerID="ed73651922035a6c95887bf08d65659d023393185c424b9ada72c40e1c23d519"
Nov 02 02:21:39.611683 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:39.611650    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-j8msj"
Nov 02 02:21:39.611889 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:39.611687    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-j8msj" event={"ID":"ea3a2c94-a15c-4214-8485-04316281816f","Type":"ContainerDied","Data":"ec2ce40dd5672a0e29e4dbf62a0a5a82fa36268cf2caacce6ed4ac42040784d1"}
Nov 02 02:21:39.611889 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:39.611778    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec2ce40dd5672a0e29e4dbf62a0a5a82fa36268cf2caacce6ed4ac42040784d1"
Nov 02 02:21:45.916434 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:45.916399    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-97pl6"]
Nov 02 02:21:45.917054 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:45.916866    5639 cpu_manager.go:414] "containerMap: removing container" podUID="ea3a2c94-a15c-4214-8485-04316281816f" containerName="revoke-keys"
Nov 02 02:21:45.917054 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:45.916878    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea3a2c94-a15c-4214-8485-04316281816f" containerName="revoke-keys"
Nov 02 02:21:45.917054 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:45.916962    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea3a2c94-a15c-4214-8485-04316281816f" containerName="revoke-keys"
Nov 02 02:21:45.921142 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:45.921124    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-97pl6"
Nov 02 02:21:45.945942 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:45.945912    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/119c417e-f2c6-408b-9463-d451b1922653-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-97pl6\" (UID: \"119c417e-f2c6-408b-9463-d451b1922653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-97pl6"
Nov 02 02:21:45.946131 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:45.946010    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/119c417e-f2c6-408b-9463-d451b1922653-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-97pl6\" (UID: \"119c417e-f2c6-408b-9463-d451b1922653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-97pl6"
Nov 02 02:21:45.946131 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:45.946056    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk4sh\" (UniqueName: \"kubernetes.io/projected/119c417e-f2c6-408b-9463-d451b1922653-kube-api-access-jk4sh\") pod \"maas-api-revoke-sub-5a29efb3-97pl6\" (UID: \"119c417e-f2c6-408b-9463-d451b1922653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-97pl6"
Nov 02 02:21:45.975080 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:45.975044    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:21:46.025045 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:46.025008    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:21:46.025045 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:46.025018    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:21:46.025344 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:46.025100    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:21:46.027252 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:46.027225    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-97pl6"]
Nov 02 02:21:46.047575 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:46.047545    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/119c417e-f2c6-408b-9463-d451b1922653-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-97pl6\" (UID: \"119c417e-f2c6-408b-9463-d451b1922653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-97pl6"
Nov 02 02:21:46.047707 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:46.047592    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/119c417e-f2c6-408b-9463-d451b1922653-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-97pl6\" (UID: \"119c417e-f2c6-408b-9463-d451b1922653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-97pl6"
Nov 02 02:21:46.047707 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:46.047650    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk4sh\" (UniqueName: \"kubernetes.io/projected/119c417e-f2c6-408b-9463-d451b1922653-kube-api-access-jk4sh\") pod \"maas-api-revoke-sub-5a29efb3-97pl6\" (UID: \"119c417e-f2c6-408b-9463-d451b1922653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-97pl6"
Nov 02 02:21:46.048382 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:46.048361    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/119c417e-f2c6-408b-9463-d451b1922653-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-97pl6\" (UID: \"119c417e-f2c6-408b-9463-d451b1922653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-97pl6"
Nov 02 02:21:46.050234 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:46.050214    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/119c417e-f2c6-408b-9463-d451b1922653-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-97pl6\" (UID: \"119c417e-f2c6-408b-9463-d451b1922653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-97pl6"
Nov 02 02:21:46.109231 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:46.109204    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk4sh\" (UniqueName: \"kubernetes.io/projected/119c417e-f2c6-408b-9463-d451b1922653-kube-api-access-jk4sh\") pod \"maas-api-revoke-sub-5a29efb3-97pl6\" (UID: \"119c417e-f2c6-408b-9463-d451b1922653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-97pl6"
Nov 02 02:21:46.234084 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:46.233994    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-97pl6"
Nov 02 02:21:46.609468 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:46.609435    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-97pl6"]
Nov 02 02:21:46.611694 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:21:46.611664    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod119c417e_f2c6_408b_9463_d451b1922653.slice/crio-c05a00b86359b0a3fa59b740493b1b600571de4c24e616d412b8cd3b0031f6f9 WatchSource:0}: Error finding container c05a00b86359b0a3fa59b740493b1b600571de4c24e616d412b8cd3b0031f6f9: Status 404 returned error can't find the container with id c05a00b86359b0a3fa59b740493b1b600571de4c24e616d412b8cd3b0031f6f9
Nov 02 02:21:46.647099 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:46.647071    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-97pl6" event={"ID":"119c417e-f2c6-408b-9463-d451b1922653","Type":"ContainerStarted","Data":"c05a00b86359b0a3fa59b740493b1b600571de4c24e616d412b8cd3b0031f6f9"}
Nov 02 02:21:47.653524 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:47.653488    5639 generic.go:356] "Generic (PLEG): container finished" podID="119c417e-f2c6-408b-9463-d451b1922653" containerID="4dfb1cbaa982e6808f3d6ed3b53505c057df14b361d49c3957d6f852e9a781bf" exitCode=0
Nov 02 02:21:47.653997 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:47.653621    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-97pl6" event={"ID":"119c417e-f2c6-408b-9463-d451b1922653","Type":"ContainerDied","Data":"4dfb1cbaa982e6808f3d6ed3b53505c057df14b361d49c3957d6f852e9a781bf"}
Nov 02 02:21:48.106137 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:48.106104    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:21:48.875486 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:48.875451    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-97pl6"
Nov 02 02:21:48.978814 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:48.978709    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/119c417e-f2c6-408b-9463-d451b1922653-kube-api-access-jk4sh\" (UniqueName: \"kubernetes.io/projected/119c417e-f2c6-408b-9463-d451b1922653-kube-api-access-jk4sh\") pod \"119c417e-f2c6-408b-9463-d451b1922653\" (UID: \"119c417e-f2c6-408b-9463-d451b1922653\") "
Nov 02 02:21:48.979022 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:48.978891    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/119c417e-f2c6-408b-9463-d451b1922653-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/119c417e-f2c6-408b-9463-d451b1922653-maas-api-openshift-ca\") pod \"119c417e-f2c6-408b-9463-d451b1922653\" (UID: \"119c417e-f2c6-408b-9463-d451b1922653\") "
Nov 02 02:21:48.979022 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:48.978949    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/119c417e-f2c6-408b-9463-d451b1922653-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/119c417e-f2c6-408b-9463-d451b1922653-maas-api-xks-ca\") pod \"119c417e-f2c6-408b-9463-d451b1922653\" (UID: \"119c417e-f2c6-408b-9463-d451b1922653\") "
Nov 02 02:21:48.979842 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:48.979809    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/119c417e-f2c6-408b-9463-d451b1922653-maas-api-openshift-ca" pod "119c417e-f2c6-408b-9463-d451b1922653" (UID: "119c417e-f2c6-408b-9463-d451b1922653"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:21:48.981754 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:48.981722    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119c417e-f2c6-408b-9463-d451b1922653-kube-api-access-jk4sh" pod "119c417e-f2c6-408b-9463-d451b1922653" (UID: "119c417e-f2c6-408b-9463-d451b1922653"). InnerVolumeSpecName "kube-api-access-jk4sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:21:48.986422 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:48.986390    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/119c417e-f2c6-408b-9463-d451b1922653-maas-api-xks-ca" pod "119c417e-f2c6-408b-9463-d451b1922653" (UID: "119c417e-f2c6-408b-9463-d451b1922653"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:21:49.080572 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:49.080523    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/119c417e-f2c6-408b-9463-d451b1922653-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:21:49.080572 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:49.080568    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk4sh\" (UniqueName: \"kubernetes.io/projected/119c417e-f2c6-408b-9463-d451b1922653-kube-api-access-jk4sh\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:21:49.080879 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:49.080584    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/119c417e-f2c6-408b-9463-d451b1922653-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:21:49.681260 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:49.681219    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-97pl6" event={"ID":"119c417e-f2c6-408b-9463-d451b1922653","Type":"ContainerDied","Data":"c05a00b86359b0a3fa59b740493b1b600571de4c24e616d412b8cd3b0031f6f9"}
Nov 02 02:21:49.681867 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:49.681842    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c05a00b86359b0a3fa59b740493b1b600571de4c24e616d412b8cd3b0031f6f9"
Nov 02 02:21:49.682070 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:49.681651    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-97pl6"
Nov 02 02:21:57.420656 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:21:57.420616    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:21:58.663702 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:21:58.663660    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:22:02.985493 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:02.985459    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j45c6"]
Nov 02 02:22:03.035225 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:03.035195    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j45c6"]
Nov 02 02:22:03.094157 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:03.094120    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nprts"]
Nov 02 02:22:03.094603 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:03.094589    5639 cpu_manager.go:414] "containerMap: removing container" podUID="119c417e-f2c6-408b-9463-d451b1922653" containerName="revoke-keys"
Nov 02 02:22:03.094669 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:03.094605    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="119c417e-f2c6-408b-9463-d451b1922653" containerName="revoke-keys"
Nov 02 02:22:03.094728 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:03.094688    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="119c417e-f2c6-408b-9463-d451b1922653" containerName="revoke-keys"
Nov 02 02:22:03.098144 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:03.098127    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nprts"
Nov 02 02:22:03.152591 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:03.152563    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:22:03.152844 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:03.152563    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:22:03.152985 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:03.152565    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:22:03.152985 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:03.152641    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:22:03.155199 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:03.155178    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nprts"]
Nov 02 02:22:03.218644 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:03.218613    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51579b3c-5e8a-49cd-b8ab-01484d7eecbb" path="/var/lib/kubelet/pods/51579b3c-5e8a-49cd-b8ab-01484d7eecbb/volumes"
Nov 02 02:22:03.257017 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:03.256946    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dt4x\" (UniqueName: \"kubernetes.io/projected/cd9e71a7-b87a-4a05-8857-9dc35de0a064-kube-api-access-5dt4x\") pod \"maas-api-revoke-sub-3d4156f1-nprts\" (UID: \"cd9e71a7-b87a-4a05-8857-9dc35de0a064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nprts"
Nov 02 02:22:03.257017 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:03.257010    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd9e71a7-b87a-4a05-8857-9dc35de0a064-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nprts\" (UID: \"cd9e71a7-b87a-4a05-8857-9dc35de0a064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nprts"
Nov 02 02:22:03.257222 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:03.257104    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd9e71a7-b87a-4a05-8857-9dc35de0a064-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nprts\" (UID: \"cd9e71a7-b87a-4a05-8857-9dc35de0a064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nprts"
Nov 02 02:22:03.358617 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:03.358583    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd9e71a7-b87a-4a05-8857-9dc35de0a064-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nprts\" (UID: \"cd9e71a7-b87a-4a05-8857-9dc35de0a064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nprts"
Nov 02 02:22:03.358769 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:03.358637    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd9e71a7-b87a-4a05-8857-9dc35de0a064-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nprts\" (UID: \"cd9e71a7-b87a-4a05-8857-9dc35de0a064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nprts"
Nov 02 02:22:03.358769 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:03.358751    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dt4x\" (UniqueName: \"kubernetes.io/projected/cd9e71a7-b87a-4a05-8857-9dc35de0a064-kube-api-access-5dt4x\") pod \"maas-api-revoke-sub-3d4156f1-nprts\" (UID: \"cd9e71a7-b87a-4a05-8857-9dc35de0a064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nprts"
Nov 02 02:22:03.359254 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:03.359236    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd9e71a7-b87a-4a05-8857-9dc35de0a064-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nprts\" (UID: \"cd9e71a7-b87a-4a05-8857-9dc35de0a064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nprts"
Nov 02 02:22:03.361156 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:03.361134    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd9e71a7-b87a-4a05-8857-9dc35de0a064-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nprts\" (UID: \"cd9e71a7-b87a-4a05-8857-9dc35de0a064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nprts"
Nov 02 02:22:03.420310 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:03.420256    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dt4x\" (UniqueName: \"kubernetes.io/projected/cd9e71a7-b87a-4a05-8857-9dc35de0a064-kube-api-access-5dt4x\") pod \"maas-api-revoke-sub-3d4156f1-nprts\" (UID: \"cd9e71a7-b87a-4a05-8857-9dc35de0a064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nprts"
Nov 02 02:22:03.719510 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:03.719480    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nprts"
Nov 02 02:22:04.093742 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:04.093716    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nprts"]
Nov 02 02:22:04.096048 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:22:04.096012    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9e71a7_b87a_4a05_8857_9dc35de0a064.slice/crio-490ba71ac61e88fa8c718a8081a55ea7d2bf3b81a5868945f1ef08e2ce4ac995 WatchSource:0}: Error finding container 490ba71ac61e88fa8c718a8081a55ea7d2bf3b81a5868945f1ef08e2ce4ac995: Status 404 returned error can't find the container with id 490ba71ac61e88fa8c718a8081a55ea7d2bf3b81a5868945f1ef08e2ce4ac995
Nov 02 02:22:04.775341 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:04.775235    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nprts" event={"ID":"cd9e71a7-b87a-4a05-8857-9dc35de0a064","Type":"ContainerStarted","Data":"d558876c79c4424b0709fdbf1ca358c2e3621effd2af89145a5b118446b3ee1b"}
Nov 02 02:22:04.775341 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:04.775317    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nprts" event={"ID":"cd9e71a7-b87a-4a05-8857-9dc35de0a064","Type":"ContainerStarted","Data":"490ba71ac61e88fa8c718a8081a55ea7d2bf3b81a5868945f1ef08e2ce4ac995"}
Nov 02 02:22:04.902426 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:04.902373    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nprts" podStartSLOduration=1.902358526 podStartE2EDuration="1.902358526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:22:04.901383384 +0000 UTC m=+3389.805065499" watchObservedRunningTime="2026-11-02 02:22:04.902358526 +0000 UTC m=+3389.806040618"
Nov 02 02:22:05.783547 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:05.783512    5639 generic.go:356] "Generic (PLEG): container finished" podID="cd9e71a7-b87a-4a05-8857-9dc35de0a064" containerID="d558876c79c4424b0709fdbf1ca358c2e3621effd2af89145a5b118446b3ee1b" exitCode=0
Nov 02 02:22:05.783970 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:05.783594    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nprts" event={"ID":"cd9e71a7-b87a-4a05-8857-9dc35de0a064","Type":"ContainerDied","Data":"d558876c79c4424b0709fdbf1ca358c2e3621effd2af89145a5b118446b3ee1b"}
Nov 02 02:22:06.921996 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:06.921968    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nprts"
Nov 02 02:22:06.994392 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:06.994345    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd9e71a7-b87a-4a05-8857-9dc35de0a064-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd9e71a7-b87a-4a05-8857-9dc35de0a064-maas-api-xks-ca\") pod \"cd9e71a7-b87a-4a05-8857-9dc35de0a064\" (UID: \"cd9e71a7-b87a-4a05-8857-9dc35de0a064\") "
Nov 02 02:22:06.994587 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:06.994404    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd9e71a7-b87a-4a05-8857-9dc35de0a064-kube-api-access-5dt4x\" (UniqueName: \"kubernetes.io/projected/cd9e71a7-b87a-4a05-8857-9dc35de0a064-kube-api-access-5dt4x\") pod \"cd9e71a7-b87a-4a05-8857-9dc35de0a064\" (UID: \"cd9e71a7-b87a-4a05-8857-9dc35de0a064\") "
Nov 02 02:22:06.994587 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:06.994461    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd9e71a7-b87a-4a05-8857-9dc35de0a064-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd9e71a7-b87a-4a05-8857-9dc35de0a064-maas-api-openshift-ca\") pod \"cd9e71a7-b87a-4a05-8857-9dc35de0a064\" (UID: \"cd9e71a7-b87a-4a05-8857-9dc35de0a064\") "
Nov 02 02:22:06.994824 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:06.994802    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd9e71a7-b87a-4a05-8857-9dc35de0a064-maas-api-openshift-ca" pod "cd9e71a7-b87a-4a05-8857-9dc35de0a064" (UID: "cd9e71a7-b87a-4a05-8857-9dc35de0a064"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:22:06.996434 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:06.996414    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd9e71a7-b87a-4a05-8857-9dc35de0a064-maas-api-xks-ca" pod "cd9e71a7-b87a-4a05-8857-9dc35de0a064" (UID: "cd9e71a7-b87a-4a05-8857-9dc35de0a064"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:22:06.996541 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:06.996521    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9e71a7-b87a-4a05-8857-9dc35de0a064-kube-api-access-5dt4x" pod "cd9e71a7-b87a-4a05-8857-9dc35de0a064" (UID: "cd9e71a7-b87a-4a05-8857-9dc35de0a064"). InnerVolumeSpecName "kube-api-access-5dt4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:22:07.089554 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:07.089518    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8v2f6"]
Nov 02 02:22:07.091312 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:07.091288    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8v2f6"]
Nov 02 02:22:07.095832 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:07.095807    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd9e71a7-b87a-4a05-8857-9dc35de0a064-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:22:07.095943 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:07.095834    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dt4x\" (UniqueName: \"kubernetes.io/projected/cd9e71a7-b87a-4a05-8857-9dc35de0a064-kube-api-access-5dt4x\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:22:07.095943 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:07.095852    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd9e71a7-b87a-4a05-8857-9dc35de0a064-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:22:07.225175 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:07.225139    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8769577d-5dcc-4bd3-8735-3e25e2db4a25" path="/var/lib/kubelet/pods/8769577d-5dcc-4bd3-8735-3e25e2db4a25/volumes"
Nov 02 02:22:07.795608 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:07.795580    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nprts"
Nov 02 02:22:07.795608 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:07.795599    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nprts" event={"ID":"cd9e71a7-b87a-4a05-8857-9dc35de0a064","Type":"ContainerDied","Data":"490ba71ac61e88fa8c718a8081a55ea7d2bf3b81a5868945f1ef08e2ce4ac995"}
Nov 02 02:22:07.795904 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:07.795641    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="490ba71ac61e88fa8c718a8081a55ea7d2bf3b81a5868945f1ef08e2ce4ac995"
Nov 02 02:22:17.694960 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:17.694926    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:22:19.048046 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:19.047980    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4sgcz"]
Nov 02 02:22:19.111307 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:19.102704    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4sgcz"]
Nov 02 02:22:19.246318 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:19.235938    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4" path="/var/lib/kubelet/pods/e5b9fec5-2f5c-4d28-adb8-65e2f4f0e7f4/volumes"
Nov 02 02:22:19.256542 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:19.256490    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:22:19.256760 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:19.256592    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx" podUID="db2e4537-de21-48f8-858e-1ead87636161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:22:27.421712 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:22:27.421673    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:22:28.189689 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:28.189655    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:22:35.414484 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.414449    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"]
Nov 02 02:22:35.414893 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.414881    5639 cpu_manager.go:414] "containerMap: removing container" podUID="cd9e71a7-b87a-4a05-8857-9dc35de0a064" containerName="revoke-keys"
Nov 02 02:22:35.414945 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.414895    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd9e71a7-b87a-4a05-8857-9dc35de0a064" containerName="revoke-keys"
Nov 02 02:22:35.414982 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.414975    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd9e71a7-b87a-4a05-8857-9dc35de0a064" containerName="revoke-keys"
Nov 02 02:22:35.418525 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.418505    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.473606 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.473575    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-fa3747-openshift-default-dockercfg-f88bm\""
Nov 02 02:22:35.510944 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.510911    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-workload-socket\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.511144 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.510962    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-data\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.511144 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.510993    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-podinfo\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.511294 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.511145    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-credential-socket\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.511294 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.511209    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-workload-certs\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.511294 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.511273    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-token\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.511430 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.511379    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blz2s\" (UniqueName: \"kubernetes.io/projected/9e865a45-c3b0-4163-bf86-73f839f3cb8d-kube-api-access-blz2s\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.511430 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.511418    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istiod-ca-cert\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.511523 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.511445    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-envoy\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.522426 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.522379    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"]
Nov 02 02:22:35.612107 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.612072    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istiod-ca-cert\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.612107 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.612110    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-envoy\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.612395 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.612166    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-workload-socket\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.612395 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.612191    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-data\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.612395 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.612208    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-podinfo\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.612395 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.612237    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-credential-socket\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.612395 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.612293    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-workload-certs\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.612395 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.612330    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-token\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.612395 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.612397    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blz2s\" (UniqueName: \"kubernetes.io/projected/9e865a45-c3b0-4163-bf86-73f839f3cb8d-kube-api-access-blz2s\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.612856 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.612802    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-credential-socket\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.613001 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.612906    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-workload-certs\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.613001 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.612982    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-data\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.613122 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.613069    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-workload-socket\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.613264 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.613242    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istiod-ca-cert\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.615398 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.615372    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-envoy\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.615562 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.615545    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-podinfo\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.676149 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.676072    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-token\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.676339 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.676246    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blz2s\" (UniqueName: \"kubernetes.io/projected/9e865a45-c3b0-4163-bf86-73f839f3cb8d-kube-api-access-blz2s\") pod \"e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") " pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:35.731008 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:35.730968    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:36.169508 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:36.169478    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"]
Nov 02 02:22:36.171713 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:22:36.171688    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e865a45_c3b0_4163_bf86_73f839f3cb8d.slice/crio-bc4d59b9ee96e7f2cd4c58342c9c54c85ca96d80306abac22f56191004a94bc4 WatchSource:0}: Error finding container bc4d59b9ee96e7f2cd4c58342c9c54c85ca96d80306abac22f56191004a94bc4: Status 404 returned error can't find the container with id bc4d59b9ee96e7f2cd4c58342c9c54c85ca96d80306abac22f56191004a94bc4
Nov 02 02:22:36.173794 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:36.173761    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:22:36.173906 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:36.173836    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:22:36.173906 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:36.173881    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:22:36.985695 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:36.985659    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq" event={"ID":"9e865a45-c3b0-4163-bf86-73f839f3cb8d","Type":"ContainerStarted","Data":"37f3ad868bf56a1dc64821863283742e014dbb0dae5e790508e70825d228d306"}
Nov 02 02:22:36.986128 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:36.985748    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq" event={"ID":"9e865a45-c3b0-4163-bf86-73f839f3cb8d","Type":"ContainerStarted","Data":"bc4d59b9ee96e7f2cd4c58342c9c54c85ca96d80306abac22f56191004a94bc4"}
Nov 02 02:22:37.106580 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:37.106524    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq" podStartSLOduration=2.106510879 podStartE2EDuration="2.106510879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:22:37.104689785 +0000 UTC m=+3422.008371912" watchObservedRunningTime="2026-11-02 02:22:37.106510879 +0000 UTC m=+3422.010192993"
Nov 02 02:22:37.731598 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:37.731564    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:37.736792 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:37.736765    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:37.991714 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:37.991637    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:37.992545 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:37.992527    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:22:39.458479 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:39.458444    5639 scope.go:122] "RemoveContainer" containerID="002b2b0a17d730268ce3554a74804716acc6428376e87b3a288e453d781dee5b"
Nov 02 02:22:52.440249 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:52.440173    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf"]
Nov 02 02:22:52.445454 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:52.445432    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf"
Nov 02 02:22:52.500388 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:52.500359    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-fa3747\""
Nov 02 02:22:52.549908 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:52.549873    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-7302d52f-dockercfg-9nwrd\""
Nov 02 02:22:52.551992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:52.551968    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf"]
Nov 02 02:22:52.598680 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:52.598644    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxt7g\" (UniqueName: \"kubernetes.io/projected/00036f2e-46c1-46c9-b9b6-d987a40f1c77-kube-api-access-bxt7g\") pod \"payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf\" (UID: \"00036f2e-46c1-46c9-b9b6-d987a40f1c77\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf"
Nov 02 02:22:52.598851 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:52.598690    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00036f2e-46c1-46c9-b9b6-d987a40f1c77-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf\" (UID: \"00036f2e-46c1-46c9-b9b6-d987a40f1c77\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf"
Nov 02 02:22:52.609096 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:52.609063    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp"]
Nov 02 02:22:52.612815 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:52.612791    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp"
Nov 02 02:22:52.670549 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:52.670514    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp"]
Nov 02 02:22:52.700324 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:52.700203    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a664188c-3afb-41b4-a941-ae65055f8874-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp\" (UID: \"a664188c-3afb-41b4-a941-ae65055f8874\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp"
Nov 02 02:22:52.700324 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:52.700265    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxt7g\" (UniqueName: \"kubernetes.io/projected/00036f2e-46c1-46c9-b9b6-d987a40f1c77-kube-api-access-bxt7g\") pod \"payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf\" (UID: \"00036f2e-46c1-46c9-b9b6-d987a40f1c77\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf"
Nov 02 02:22:52.700324 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:52.700313    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00036f2e-46c1-46c9-b9b6-d987a40f1c77-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf\" (UID: \"00036f2e-46c1-46c9-b9b6-d987a40f1c77\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf"
Nov 02 02:22:52.700645 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:52.700411    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x24n9\" (UniqueName: \"kubernetes.io/projected/a664188c-3afb-41b4-a941-ae65055f8874-kube-api-access-x24n9\") pod \"payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp\" (UID: \"a664188c-3afb-41b4-a941-ae65055f8874\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp"
Nov 02 02:22:52.701011 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:52.700987    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00036f2e-46c1-46c9-b9b6-d987a40f1c77-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf\" (UID: \"00036f2e-46c1-46c9-b9b6-d987a40f1c77\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf"
Nov 02 02:22:52.761653 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:52.761622    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxt7g\" (UniqueName: \"kubernetes.io/projected/00036f2e-46c1-46c9-b9b6-d987a40f1c77-kube-api-access-bxt7g\") pod \"payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf\" (UID: \"00036f2e-46c1-46c9-b9b6-d987a40f1c77\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf"
Nov 02 02:22:52.801194 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:52.801157    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x24n9\" (UniqueName: \"kubernetes.io/projected/a664188c-3afb-41b4-a941-ae65055f8874-kube-api-access-x24n9\") pod \"payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp\" (UID: \"a664188c-3afb-41b4-a941-ae65055f8874\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp"
Nov 02 02:22:52.801398 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:52.801207    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a664188c-3afb-41b4-a941-ae65055f8874-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp\" (UID: \"a664188c-3afb-41b4-a941-ae65055f8874\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp"
Nov 02 02:22:52.801918 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:52.801897    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a664188c-3afb-41b4-a941-ae65055f8874-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp\" (UID: \"a664188c-3afb-41b4-a941-ae65055f8874\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp"
Nov 02 02:22:52.865178 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:52.865144    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x24n9\" (UniqueName: \"kubernetes.io/projected/a664188c-3afb-41b4-a941-ae65055f8874-kube-api-access-x24n9\") pod \"payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp\" (UID: \"a664188c-3afb-41b4-a941-ae65055f8874\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp"
Nov 02 02:22:52.926078 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:52.926042    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp"
Nov 02 02:22:53.059003 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:53.058965    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf"
Nov 02 02:22:53.301313 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:53.301264    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp"]
Nov 02 02:22:53.303349 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:22:53.303322    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda664188c_3afb_41b4_a941_ae65055f8874.slice/crio-609e36cd685ed4264a81acff3de8aef5b1854fb9b6f2baf96028a14afad1ae20 WatchSource:0}: Error finding container 609e36cd685ed4264a81acff3de8aef5b1854fb9b6f2baf96028a14afad1ae20: Status 404 returned error can't find the container with id 609e36cd685ed4264a81acff3de8aef5b1854fb9b6f2baf96028a14afad1ae20
Nov 02 02:22:53.439675 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:53.435853    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf"]
Nov 02 02:22:53.443622 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:22:53.443596    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00036f2e_46c1_46c9_b9b6_d987a40f1c77.slice/crio-9c19d026c714b303a94ff9d0053772978b0eb91ac5f1f59c1dc7188a2f865ec9 WatchSource:0}: Error finding container 9c19d026c714b303a94ff9d0053772978b0eb91ac5f1f59c1dc7188a2f865ec9: Status 404 returned error can't find the container with id 9c19d026c714b303a94ff9d0053772978b0eb91ac5f1f59c1dc7188a2f865ec9
Nov 02 02:22:54.079508 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:54.079469    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp" event={"ID":"a664188c-3afb-41b4-a941-ae65055f8874","Type":"ContainerStarted","Data":"4099ecb2abec0d4f35afd5702482cabdd68f6c8a035c7f996504bad2eb2bbcd8"}
Nov 02 02:22:54.079746 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:54.079533    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp" event={"ID":"a664188c-3afb-41b4-a941-ae65055f8874","Type":"ContainerStarted","Data":"609e36cd685ed4264a81acff3de8aef5b1854fb9b6f2baf96028a14afad1ae20"}
Nov 02 02:22:54.079746 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:54.079562    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp"
Nov 02 02:22:54.081194 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:54.081167    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf" event={"ID":"00036f2e-46c1-46c9-b9b6-d987a40f1c77","Type":"ContainerStarted","Data":"bed339ff41457fb83e0b104b9147bf2caaa37973dac585c252f5255315c6fc7e"}
Nov 02 02:22:54.081372 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:54.081219    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf" event={"ID":"00036f2e-46c1-46c9-b9b6-d987a40f1c77","Type":"ContainerStarted","Data":"9c19d026c714b303a94ff9d0053772978b0eb91ac5f1f59c1dc7188a2f865ec9"}
Nov 02 02:22:54.081372 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:54.081265    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf"
Nov 02 02:22:54.198761 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:54.198696    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp" podStartSLOduration=2.198678013 podStartE2EDuration="2.198678013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:22:54.19654142 +0000 UTC m=+3439.100223533" watchObservedRunningTime="2026-11-02 02:22:54.198678013 +0000 UTC m=+3439.102360128"
Nov 02 02:22:54.281293 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:54.281250    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:22:54.321684 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:54.321608    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf" podStartSLOduration=2.321588758 podStartE2EDuration="2.321588758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:22:54.313673361 +0000 UTC m=+3439.217355478" watchObservedRunningTime="2026-11-02 02:22:54.321588758 +0000 UTC m=+3439.225270873"
Nov 02 02:22:55.047637 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:55.047599    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nkhnx"]
Nov 02 02:22:55.096884 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:55.096849    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nkhnx"]
Nov 02 02:22:55.219212 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:22:55.219173    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22242546-a0a6-4a08-872b-0e3cd4d57689" path="/var/lib/kubelet/pods/22242546-a0a6-4a08-872b-0e3cd4d57689/volumes"
Nov 02 02:22:57.420864 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:22:57.420816    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:23:04.142656 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:04.142621    5639 generic.go:356] "Generic (PLEG): container finished" podID="9e865a45-c3b0-4163-bf86-73f839f3cb8d" containerID="37f3ad868bf56a1dc64821863283742e014dbb0dae5e790508e70825d228d306" exitCode=0
Nov 02 02:23:04.143390 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:04.142696    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq" event={"ID":"9e865a45-c3b0-4163-bf86-73f839f3cb8d","Type":"ContainerDied","Data":"37f3ad868bf56a1dc64821863283742e014dbb0dae5e790508e70825d228d306"}
Nov 02 02:23:04.143390 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:04.143060    5639 scope.go:122] "RemoveContainer" containerID="37f3ad868bf56a1dc64821863283742e014dbb0dae5e790508e70825d228d306"
Nov 02 02:23:04.144198 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:04.144168    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:23:04.144255 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:04.144246    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:23:04.144319 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:04.144299    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:23:05.093456 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:05.093427    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf"
Nov 02 02:23:05.093639 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:05.093504    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp"
Nov 02 02:23:05.150811 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:05.150778    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq" event={"ID":"9e865a45-c3b0-4163-bf86-73f839f3cb8d","Type":"ContainerStarted","Data":"81d117c1b322d3f8a82f38ddbe8738fffed890f95174fbb3fc385e0e90299658"}
Nov 02 02:23:05.731458 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:05.731419    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:23:05.732653 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:05.732631    5639 patch_prober.go:29] interesting pod/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body=
Nov 02 02:23:05.732715 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:05.732697    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq" podUID="9e865a45-c3b0-4163-bf86-73f839f3cb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused"
Nov 02 02:23:06.048250 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:06.048161    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n7xh7"]
Nov 02 02:23:06.055417 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:06.055386    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n7xh7"]
Nov 02 02:23:06.731921 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:06.731882    5639 patch_prober.go:29] interesting pod/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body=
Nov 02 02:23:06.732473 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:06.731948    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq" podUID="9e865a45-c3b0-4163-bf86-73f839f3cb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused"
Nov 02 02:23:07.217212 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:07.217175    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40123de6-f019-4408-bed9-8685479a3ff0" path="/var/lib/kubelet/pods/40123de6-f019-4408-bed9-8685479a3ff0/volumes"
Nov 02 02:23:07.731904 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:07.731875    5639 patch_prober.go:29] interesting pod/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body=
Nov 02 02:23:07.732093 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:07.731923    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq" podUID="9e865a45-c3b0-4163-bf86-73f839f3cb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused"
Nov 02 02:23:07.991749 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:07.991662    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:23:08.731613 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:08.731573    5639 patch_prober.go:29] interesting pod/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body=
Nov 02 02:23:08.731846 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:08.731636    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq" podUID="9e865a45-c3b0-4163-bf86-73f839f3cb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused"
Nov 02 02:23:09.731681 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:09.731635    5639 patch_prober.go:29] interesting pod/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body=
Nov 02 02:23:09.732110 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:09.731697    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq" podUID="9e865a45-c3b0-4163-bf86-73f839f3cb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused"
Nov 02 02:23:09.917044 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:09.917011    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"]
Nov 02 02:23:09.921007 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:09.920980    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"
Nov 02 02:23:09.986826 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:09.986731    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-fa3747\"/\"openshift-service-ca.crt\""
Nov 02 02:23:10.022943 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:10.022904    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-fa3747\"/\"kube-root-ca.crt\""
Nov 02 02:23:10.023223 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:10.023202    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-model-cache\") pod \"simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w\" (UID: \"57757f1a-681c-436e-93a6-652ab4ec0624\") " pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"
Nov 02 02:23:10.023290 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:10.023251    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7s6g\" (UniqueName: \"kubernetes.io/projected/57757f1a-681c-436e-93a6-652ab4ec0624-kube-api-access-l7s6g\") pod \"simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w\" (UID: \"57757f1a-681c-436e-93a6-652ab4ec0624\") " pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"
Nov 02 02:23:10.023394 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:10.023377    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-home\") pod \"simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w\" (UID: \"57757f1a-681c-436e-93a6-652ab4ec0624\") " pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"
Nov 02 02:23:10.023445 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:10.023402    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-dshm\") pod \"simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w\" (UID: \"57757f1a-681c-436e-93a6-652ab4ec0624\") " pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"
Nov 02 02:23:10.023521 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:10.023503    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-tmp-dir\") pod \"simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w\" (UID: \"57757f1a-681c-436e-93a6-652ab4ec0624\") " pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"
Nov 02 02:23:10.023562 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:10.023540    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57757f1a-681c-436e-93a6-652ab4ec0624-tls-certs\") pod \"simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w\" (UID: \"57757f1a-681c-436e-93a6-652ab4ec0624\") " pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"
Nov 02 02:23:10.036536 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:10.036502    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-fa3747\"/\"default-dockercfg-2d579\""
Nov 02 02:23:10.036734 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:10.036545    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-fa3747\"/\"simulator-single-main-fa3747-kserve-self-signed-certs\""
Nov 02 02:23:10.038585 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:10.038560    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"]
Nov 02 02:23:10.124596 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:10.124565    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-home\") pod \"simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w\" (UID: \"57757f1a-681c-436e-93a6-652ab4ec0624\") " pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"
Nov 02 02:23:10.124596 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:10.124598    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-dshm\") pod \"simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w\" (UID: \"57757f1a-681c-436e-93a6-652ab4ec0624\") " pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"
Nov 02 02:23:10.124893 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:10.124645    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-tmp-dir\") pod \"simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w\" (UID: \"57757f1a-681c-436e-93a6-652ab4ec0624\") " pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"
Nov 02 02:23:10.124893 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:10.124667    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57757f1a-681c-436e-93a6-652ab4ec0624-tls-certs\") pod \"simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w\" (UID: \"57757f1a-681c-436e-93a6-652ab4ec0624\") " pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"
Nov 02 02:23:10.124893 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:10.124753    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-model-cache\") pod \"simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w\" (UID: \"57757f1a-681c-436e-93a6-652ab4ec0624\") " pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"
Nov 02 02:23:10.124893 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:10.124816    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7s6g\" (UniqueName: \"kubernetes.io/projected/57757f1a-681c-436e-93a6-652ab4ec0624-kube-api-access-l7s6g\") pod \"simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w\" (UID: \"57757f1a-681c-436e-93a6-652ab4ec0624\") " pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"
Nov 02 02:23:10.125145 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:10.125124    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-tmp-dir\") pod \"simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w\" (UID: \"57757f1a-681c-436e-93a6-652ab4ec0624\") " pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"
Nov 02 02:23:10.125210 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:10.125184    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-home\") pod \"simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w\" (UID: \"57757f1a-681c-436e-93a6-652ab4ec0624\") " pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"
Nov 02 02:23:10.125313 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:10.125265    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-model-cache\") pod \"simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w\" (UID: \"57757f1a-681c-436e-93a6-652ab4ec0624\") " pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"
Nov 02 02:23:10.127158 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:10.127127    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-dshm\") pod \"simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w\" (UID: \"57757f1a-681c-436e-93a6-652ab4ec0624\") " pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"
Nov 02 02:23:10.127483 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:10.127463    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57757f1a-681c-436e-93a6-652ab4ec0624-tls-certs\") pod \"simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w\" (UID: \"57757f1a-681c-436e-93a6-652ab4ec0624\") " pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"
Nov 02 02:23:10.189574 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:10.189538    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7s6g\" (UniqueName: \"kubernetes.io/projected/57757f1a-681c-436e-93a6-652ab4ec0624-kube-api-access-l7s6g\") pod \"simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w\" (UID: \"57757f1a-681c-436e-93a6-652ab4ec0624\") " pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"
Nov 02 02:23:10.233842 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:10.233795    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"
Nov 02 02:23:10.637807 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:10.629895    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"]
Nov 02 02:23:10.736068 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:10.736038    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:23:10.737004 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:10.736986    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:23:11.208794 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:11.208759    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w" event={"ID":"57757f1a-681c-436e-93a6-652ab4ec0624","Type":"ContainerStarted","Data":"6edd2b954bf13e6ab653b2a9ace90dd4ee3fb96fd55b3ca2e02a15c6a4a03c07"}
Nov 02 02:23:11.208981 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:11.208803    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w" event={"ID":"57757f1a-681c-436e-93a6-652ab4ec0624","Type":"ContainerStarted","Data":"13f9d2ef72e09fbdc621a6fda42f069d42a39cd26711e211c758769f807c6f54"}
Nov 02 02:23:11.233957 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:11.233920    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"
Nov 02 02:23:11.327574 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:11.327519    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w" podStartSLOduration=2.327505002 podStartE2EDuration="2.327505002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:23:11.32590039 +0000 UTC m=+3456.229582504" watchObservedRunningTime="2026-11-02 02:23:11.327505002 +0000 UTC m=+3456.231187116"
Nov 02 02:23:14.677773 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:14.677737    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:23:20.234739 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:20.234698    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"
Nov 02 02:23:20.247571 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:20.247535    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"
Nov 02 02:23:20.272969 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:20.272939    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"
Nov 02 02:23:27.420690 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:23:27.420653    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:23:39.014991 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:39.014096    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:23:39.485311 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:39.485245    5639 scope.go:122] "RemoveContainer" containerID="e30be2785045befb8df3d2c5e08450da9bb0505fd2c5d0f5dac737ffbe196628"
Nov 02 02:23:39.500506 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:39.500474    5639 scope.go:122] "RemoveContainer" containerID="2e9562ea578d40b1771fed6e4fe43ef5840bd227e5d66acd5134eedcf53063a0"
Nov 02 02:23:41.048878 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:41.048842    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p66fd"]
Nov 02 02:23:41.096975 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:41.096935    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p66fd"]
Nov 02 02:23:41.218155 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:41.218121    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83668079-47fe-4dd4-82d1-227ea5b13c31" path="/var/lib/kubelet/pods/83668079-47fe-4dd4-82d1-227ea5b13c31/volumes"
Nov 02 02:23:53.347054 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:53.347022    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d57d4200-jlbtw"]
Nov 02 02:23:53.352357 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:53.352340    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57d4200-jlbtw"
Nov 02 02:23:53.455265 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:53.455220    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:23:53.455265 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:53.455241    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:23:53.455265 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:53.455236    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:23:53.455265 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:53.455220    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:23:53.457239 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:53.457220    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d57d4200-jlbtw"]
Nov 02 02:23:53.494973 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:53.494950    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc4jh\" (UniqueName: \"kubernetes.io/projected/eb2272f9-5d1c-4dc3-beaa-bdd278852801-kube-api-access-gc4jh\") pod \"maas-api-revoke-sub-d57d4200-jlbtw\" (UID: \"eb2272f9-5d1c-4dc3-beaa-bdd278852801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57d4200-jlbtw"
Nov 02 02:23:53.495096 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:53.494980    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb2272f9-5d1c-4dc3-beaa-bdd278852801-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d57d4200-jlbtw\" (UID: \"eb2272f9-5d1c-4dc3-beaa-bdd278852801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57d4200-jlbtw"
Nov 02 02:23:53.495136 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:53.495117    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb2272f9-5d1c-4dc3-beaa-bdd278852801-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d57d4200-jlbtw\" (UID: \"eb2272f9-5d1c-4dc3-beaa-bdd278852801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57d4200-jlbtw"
Nov 02 02:23:53.596513 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:53.596469    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb2272f9-5d1c-4dc3-beaa-bdd278852801-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d57d4200-jlbtw\" (UID: \"eb2272f9-5d1c-4dc3-beaa-bdd278852801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57d4200-jlbtw"
Nov 02 02:23:53.596690 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:53.596624    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc4jh\" (UniqueName: \"kubernetes.io/projected/eb2272f9-5d1c-4dc3-beaa-bdd278852801-kube-api-access-gc4jh\") pod \"maas-api-revoke-sub-d57d4200-jlbtw\" (UID: \"eb2272f9-5d1c-4dc3-beaa-bdd278852801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57d4200-jlbtw"
Nov 02 02:23:53.596690 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:53.596651    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb2272f9-5d1c-4dc3-beaa-bdd278852801-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d57d4200-jlbtw\" (UID: \"eb2272f9-5d1c-4dc3-beaa-bdd278852801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57d4200-jlbtw"
Nov 02 02:23:53.597402 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:53.597359    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb2272f9-5d1c-4dc3-beaa-bdd278852801-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d57d4200-jlbtw\" (UID: \"eb2272f9-5d1c-4dc3-beaa-bdd278852801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57d4200-jlbtw"
Nov 02 02:23:53.598945 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:53.598912    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb2272f9-5d1c-4dc3-beaa-bdd278852801-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d57d4200-jlbtw\" (UID: \"eb2272f9-5d1c-4dc3-beaa-bdd278852801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57d4200-jlbtw"
Nov 02 02:23:53.657954 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:53.657926    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc4jh\" (UniqueName: \"kubernetes.io/projected/eb2272f9-5d1c-4dc3-beaa-bdd278852801-kube-api-access-gc4jh\") pod \"maas-api-revoke-sub-d57d4200-jlbtw\" (UID: \"eb2272f9-5d1c-4dc3-beaa-bdd278852801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57d4200-jlbtw"
Nov 02 02:23:53.664861 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:53.664834    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57d4200-jlbtw"
Nov 02 02:23:54.038847 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:54.038815    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d57d4200-jlbtw"]
Nov 02 02:23:54.041834 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:23:54.041807    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2272f9_5d1c_4dc3_beaa_bdd278852801.slice/crio-56002af2909e183f6b4c6c43facd10519b3a0670397a66af19466666c29e0f9d WatchSource:0}: Error finding container 56002af2909e183f6b4c6c43facd10519b3a0670397a66af19466666c29e0f9d: Status 404 returned error can't find the container with id 56002af2909e183f6b4c6c43facd10519b3a0670397a66af19466666c29e0f9d
Nov 02 02:23:54.453522 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:54.453481    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57d4200-jlbtw" event={"ID":"eb2272f9-5d1c-4dc3-beaa-bdd278852801","Type":"ContainerStarted","Data":"49700903dcaea47328e3f0b88aa57770385a6356569239feea5b9a20df5bc44c"}
Nov 02 02:23:54.453987 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:54.453533    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57d4200-jlbtw" event={"ID":"eb2272f9-5d1c-4dc3-beaa-bdd278852801","Type":"ContainerStarted","Data":"56002af2909e183f6b4c6c43facd10519b3a0670397a66af19466666c29e0f9d"}
Nov 02 02:23:54.570956 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:54.570903    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57d4200-jlbtw" podStartSLOduration=1.570888419 podStartE2EDuration="1.570888419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:23:54.568968156 +0000 UTC m=+3499.472650285" watchObservedRunningTime="2026-11-02 02:23:54.570888419 +0000 UTC m=+3499.474570534"
Nov 02 02:23:55.460569 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:55.460525    5639 generic.go:356] "Generic (PLEG): container finished" podID="eb2272f9-5d1c-4dc3-beaa-bdd278852801" containerID="49700903dcaea47328e3f0b88aa57770385a6356569239feea5b9a20df5bc44c" exitCode=0
Nov 02 02:23:55.460569 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:55.460564    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57d4200-jlbtw" event={"ID":"eb2272f9-5d1c-4dc3-beaa-bdd278852801","Type":"ContainerDied","Data":"49700903dcaea47328e3f0b88aa57770385a6356569239feea5b9a20df5bc44c"}
Nov 02 02:23:56.609025 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:56.609002    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57d4200-jlbtw"
Nov 02 02:23:56.730019 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:56.729937    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb2272f9-5d1c-4dc3-beaa-bdd278852801-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb2272f9-5d1c-4dc3-beaa-bdd278852801-maas-api-xks-ca\") pod \"eb2272f9-5d1c-4dc3-beaa-bdd278852801\" (UID: \"eb2272f9-5d1c-4dc3-beaa-bdd278852801\") "
Nov 02 02:23:56.730207 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:56.730067    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb2272f9-5d1c-4dc3-beaa-bdd278852801-kube-api-access-gc4jh\" (UniqueName: \"kubernetes.io/projected/eb2272f9-5d1c-4dc3-beaa-bdd278852801-kube-api-access-gc4jh\") pod \"eb2272f9-5d1c-4dc3-beaa-bdd278852801\" (UID: \"eb2272f9-5d1c-4dc3-beaa-bdd278852801\") "
Nov 02 02:23:56.730207 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:56.730132    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb2272f9-5d1c-4dc3-beaa-bdd278852801-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb2272f9-5d1c-4dc3-beaa-bdd278852801-maas-api-openshift-ca\") pod \"eb2272f9-5d1c-4dc3-beaa-bdd278852801\" (UID: \"eb2272f9-5d1c-4dc3-beaa-bdd278852801\") "
Nov 02 02:23:56.730542 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:56.730517    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb2272f9-5d1c-4dc3-beaa-bdd278852801-maas-api-openshift-ca" pod "eb2272f9-5d1c-4dc3-beaa-bdd278852801" (UID: "eb2272f9-5d1c-4dc3-beaa-bdd278852801"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:23:56.732350 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:56.732321    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb2272f9-5d1c-4dc3-beaa-bdd278852801-kube-api-access-gc4jh" pod "eb2272f9-5d1c-4dc3-beaa-bdd278852801" (UID: "eb2272f9-5d1c-4dc3-beaa-bdd278852801"). InnerVolumeSpecName "kube-api-access-gc4jh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:23:56.732457 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:56.732405    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb2272f9-5d1c-4dc3-beaa-bdd278852801-maas-api-xks-ca" pod "eb2272f9-5d1c-4dc3-beaa-bdd278852801" (UID: "eb2272f9-5d1c-4dc3-beaa-bdd278852801"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:23:56.831851 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:56.831823    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb2272f9-5d1c-4dc3-beaa-bdd278852801-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:23:56.831851 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:56.831848    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc4jh\" (UniqueName: \"kubernetes.io/projected/eb2272f9-5d1c-4dc3-beaa-bdd278852801-kube-api-access-gc4jh\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:23:56.832056 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:56.831859    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb2272f9-5d1c-4dc3-beaa-bdd278852801-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:23:57.419888 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:23:57.419845    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:23:57.472731 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:57.472702    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57d4200-jlbtw"
Nov 02 02:23:57.472907 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:57.472697    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57d4200-jlbtw" event={"ID":"eb2272f9-5d1c-4dc3-beaa-bdd278852801","Type":"ContainerDied","Data":"56002af2909e183f6b4c6c43facd10519b3a0670397a66af19466666c29e0f9d"}
Nov 02 02:23:57.472907 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:23:57.472849    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56002af2909e183f6b4c6c43facd10519b3a0670397a66af19466666c29e0f9d"
Nov 02 02:24:05.547944 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:05.547900    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5"]
Nov 02 02:24:05.548548 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:05.548528    5639 cpu_manager.go:414] "containerMap: removing container" podUID="eb2272f9-5d1c-4dc3-beaa-bdd278852801" containerName="revoke-keys"
Nov 02 02:24:05.548605 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:05.548552    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb2272f9-5d1c-4dc3-beaa-bdd278852801" containerName="revoke-keys"
Nov 02 02:24:05.548652 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:05.548640    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb2272f9-5d1c-4dc3-beaa-bdd278852801" containerName="revoke-keys"
Nov 02 02:24:05.551996 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:05.551978    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5"
Nov 02 02:24:05.629256 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:05.629222    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a703dd1-e3c2-498a-b25e-6146c8192168-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5\" (UID: \"8a703dd1-e3c2-498a-b25e-6146c8192168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5"
Nov 02 02:24:05.629474 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:05.629303    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a703dd1-e3c2-498a-b25e-6146c8192168-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5\" (UID: \"8a703dd1-e3c2-498a-b25e-6146c8192168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5"
Nov 02 02:24:05.629474 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:05.629446    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz8wl\" (UniqueName: \"kubernetes.io/projected/8a703dd1-e3c2-498a-b25e-6146c8192168-kube-api-access-rz8wl\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5\" (UID: \"8a703dd1-e3c2-498a-b25e-6146c8192168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5"
Nov 02 02:24:05.655843 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:05.655806    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:24:05.656052 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:05.655854    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:24:05.656052 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:05.655945    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:24:05.656052 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:05.655882    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:24:05.658082 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:05.658060    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5"]
Nov 02 02:24:05.731103 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:05.731068    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz8wl\" (UniqueName: \"kubernetes.io/projected/8a703dd1-e3c2-498a-b25e-6146c8192168-kube-api-access-rz8wl\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5\" (UID: \"8a703dd1-e3c2-498a-b25e-6146c8192168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5"
Nov 02 02:24:05.731334 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:05.731151    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a703dd1-e3c2-498a-b25e-6146c8192168-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5\" (UID: \"8a703dd1-e3c2-498a-b25e-6146c8192168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5"
Nov 02 02:24:05.731334 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:05.731223    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a703dd1-e3c2-498a-b25e-6146c8192168-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5\" (UID: \"8a703dd1-e3c2-498a-b25e-6146c8192168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5"
Nov 02 02:24:05.731902 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:05.731873    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a703dd1-e3c2-498a-b25e-6146c8192168-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5\" (UID: \"8a703dd1-e3c2-498a-b25e-6146c8192168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5"
Nov 02 02:24:05.733724 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:05.733697    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a703dd1-e3c2-498a-b25e-6146c8192168-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5\" (UID: \"8a703dd1-e3c2-498a-b25e-6146c8192168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5"
Nov 02 02:24:05.793994 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:05.793961    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz8wl\" (UniqueName: \"kubernetes.io/projected/8a703dd1-e3c2-498a-b25e-6146c8192168-kube-api-access-rz8wl\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5\" (UID: \"8a703dd1-e3c2-498a-b25e-6146c8192168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5"
Nov 02 02:24:05.864406 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:05.864368    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5"
Nov 02 02:24:06.259946 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:06.259916    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5"]
Nov 02 02:24:06.261787 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:24:06.261754    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a703dd1_e3c2_498a_b25e_6146c8192168.slice/crio-0d2cb7d7cc48037a3889c162eea4528d7a2a92e0e35c5423ca643e7cfffe1d96 WatchSource:0}: Error finding container 0d2cb7d7cc48037a3889c162eea4528d7a2a92e0e35c5423ca643e7cfffe1d96: Status 404 returned error can't find the container with id 0d2cb7d7cc48037a3889c162eea4528d7a2a92e0e35c5423ca643e7cfffe1d96
Nov 02 02:24:06.526226 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:06.526133    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5" event={"ID":"8a703dd1-e3c2-498a-b25e-6146c8192168","Type":"ContainerStarted","Data":"0d2cb7d7cc48037a3889c162eea4528d7a2a92e0e35c5423ca643e7cfffe1d96"}
Nov 02 02:24:07.533490 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:07.533398    5639 generic.go:356] "Generic (PLEG): container finished" podID="8a703dd1-e3c2-498a-b25e-6146c8192168" containerID="21cb8e4eef9d162d3733cc240f50268e512fbd7ed37a5ec81ca15a2cd3c67338" exitCode=0
Nov 02 02:24:07.533864 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:07.533485    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5" event={"ID":"8a703dd1-e3c2-498a-b25e-6146c8192168","Type":"ContainerDied","Data":"21cb8e4eef9d162d3733cc240f50268e512fbd7ed37a5ec81ca15a2cd3c67338"}
Nov 02 02:24:08.050124 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:08.050081    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lgm9s"]
Nov 02 02:24:08.061823 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:08.061783    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lgm9s"]
Nov 02 02:24:08.209417 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:08.209373    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:24:08.685430 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:08.685405    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5"
Nov 02 02:24:08.765432 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:08.765339    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a703dd1-e3c2-498a-b25e-6146c8192168-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a703dd1-e3c2-498a-b25e-6146c8192168-maas-api-xks-ca\") pod \"8a703dd1-e3c2-498a-b25e-6146c8192168\" (UID: \"8a703dd1-e3c2-498a-b25e-6146c8192168\") "
Nov 02 02:24:08.766048 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:08.766024    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a703dd1-e3c2-498a-b25e-6146c8192168-kube-api-access-rz8wl\" (UniqueName: \"kubernetes.io/projected/8a703dd1-e3c2-498a-b25e-6146c8192168-kube-api-access-rz8wl\") pod \"8a703dd1-e3c2-498a-b25e-6146c8192168\" (UID: \"8a703dd1-e3c2-498a-b25e-6146c8192168\") "
Nov 02 02:24:08.768348 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:08.767583    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a703dd1-e3c2-498a-b25e-6146c8192168-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a703dd1-e3c2-498a-b25e-6146c8192168-maas-api-openshift-ca\") pod \"8a703dd1-e3c2-498a-b25e-6146c8192168\" (UID: \"8a703dd1-e3c2-498a-b25e-6146c8192168\") "
Nov 02 02:24:08.768781 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:08.768736    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a703dd1-e3c2-498a-b25e-6146c8192168-maas-api-openshift-ca" pod "8a703dd1-e3c2-498a-b25e-6146c8192168" (UID: "8a703dd1-e3c2-498a-b25e-6146c8192168"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:24:08.771663 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:08.771628    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a703dd1-e3c2-498a-b25e-6146c8192168-maas-api-xks-ca" pod "8a703dd1-e3c2-498a-b25e-6146c8192168" (UID: "8a703dd1-e3c2-498a-b25e-6146c8192168"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:24:08.773841 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:08.773812    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a703dd1-e3c2-498a-b25e-6146c8192168-kube-api-access-rz8wl" pod "8a703dd1-e3c2-498a-b25e-6146c8192168" (UID: "8a703dd1-e3c2-498a-b25e-6146c8192168"). InnerVolumeSpecName "kube-api-access-rz8wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:24:08.871354 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:08.871319    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a703dd1-e3c2-498a-b25e-6146c8192168-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:08.871631 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:08.871612    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz8wl\" (UniqueName: \"kubernetes.io/projected/8a703dd1-e3c2-498a-b25e-6146c8192168-kube-api-access-rz8wl\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:08.871722 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:08.871711    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a703dd1-e3c2-498a-b25e-6146c8192168-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:09.218337 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:09.218248    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1301372-aa12-416f-a565-74989f5641ff" path="/var/lib/kubelet/pods/f1301372-aa12-416f-a565-74989f5641ff/volumes"
Nov 02 02:24:09.546878 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:09.546785    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5"
Nov 02 02:24:09.546878 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:09.546816    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5" event={"ID":"8a703dd1-e3c2-498a-b25e-6146c8192168","Type":"ContainerDied","Data":"0d2cb7d7cc48037a3889c162eea4528d7a2a92e0e35c5423ca643e7cfffe1d96"}
Nov 02 02:24:09.546878 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:09.546867    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d2cb7d7cc48037a3889c162eea4528d7a2a92e0e35c5423ca643e7cfffe1d96"
Nov 02 02:24:09.975897 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:09.975864    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"]
Nov 02 02:24:17.046523 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:17.046441    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf"]
Nov 02 02:24:17.152688 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:17.152652    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp"]
Nov 02 02:24:22.047565 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:22.047510    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf" podUID="00036f2e-46c1-46c9-b9b6-d987a40f1c77" containerName="payload-processing" containerID="cri-o://bed339ff41457fb83e0b104b9147bf2caaa37973dac585c252f5255315c6fc7e" gracePeriod=25
Nov 02 02:24:22.153710 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:22.153643    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp" podUID="a664188c-3afb-41b4-a941-ae65055f8874" containerName="payload-pre-processing" containerID="cri-o://4099ecb2abec0d4f35afd5702482cabdd68f6c8a035c7f996504bad2eb2bbcd8" gracePeriod=25
Nov 02 02:24:24.678423 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:24.678382    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5"]
Nov 02 02:24:24.728043 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:24.727992    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa3747-r5fq5"]
Nov 02 02:24:24.997086 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:24.996977    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w" podUID="57757f1a-681c-436e-93a6-652ab4ec0624" containerName="main" containerID="cri-o://6edd2b954bf13e6ab653b2a9ace90dd4ee3fb96fd55b3ca2e02a15c6a4a03c07" gracePeriod=45
Nov 02 02:24:25.091586 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.091548    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp" podUID="a664188c-3afb-41b4-a941-ae65055f8874" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:24:25.091804 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.091560    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf" podUID="00036f2e-46c1-46c9-b9b6-d987a40f1c77" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:24:25.219726 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.219691    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a703dd1-e3c2-498a-b25e-6146c8192168" path="/var/lib/kubelet/pods/8a703dd1-e3c2-498a-b25e-6146c8192168/volumes"
Nov 02 02:24:25.257406 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.257326    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"
Nov 02 02:24:25.349537 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.349500    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-home\" (UniqueName: \"kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-home\") pod \"57757f1a-681c-436e-93a6-652ab4ec0624\" (UID: \"57757f1a-681c-436e-93a6-652ab4ec0624\") "
Nov 02 02:24:25.349729 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.349552    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57757f1a-681c-436e-93a6-652ab4ec0624-kube-api-access-l7s6g\" (UniqueName: \"kubernetes.io/projected/57757f1a-681c-436e-93a6-652ab4ec0624-kube-api-access-l7s6g\") pod \"57757f1a-681c-436e-93a6-652ab4ec0624\" (UID: \"57757f1a-681c-436e-93a6-652ab4ec0624\") "
Nov 02 02:24:25.349729 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.349574    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-dshm\" (UniqueName: \"kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-dshm\") pod \"57757f1a-681c-436e-93a6-652ab4ec0624\" (UID: \"57757f1a-681c-436e-93a6-652ab4ec0624\") "
Nov 02 02:24:25.349729 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.349605    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57757f1a-681c-436e-93a6-652ab4ec0624-tls-certs\" (UniqueName: \"kubernetes.io/secret/57757f1a-681c-436e-93a6-652ab4ec0624-tls-certs\") pod \"57757f1a-681c-436e-93a6-652ab4ec0624\" (UID: \"57757f1a-681c-436e-93a6-652ab4ec0624\") "
Nov 02 02:24:25.349729 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.349687    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-tmp-dir\") pod \"57757f1a-681c-436e-93a6-652ab4ec0624\" (UID: \"57757f1a-681c-436e-93a6-652ab4ec0624\") "
Nov 02 02:24:25.349903 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.349770    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-model-cache\") pod \"57757f1a-681c-436e-93a6-652ab4ec0624\" (UID: \"57757f1a-681c-436e-93a6-652ab4ec0624\") "
Nov 02 02:24:25.349903 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.349865    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-home" pod "57757f1a-681c-436e-93a6-652ab4ec0624" (UID: "57757f1a-681c-436e-93a6-652ab4ec0624"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:24:25.350143 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.350035    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-tmp-dir" pod "57757f1a-681c-436e-93a6-652ab4ec0624" (UID: "57757f1a-681c-436e-93a6-652ab4ec0624"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:24:25.350143 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.350090    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-model-cache" pod "57757f1a-681c-436e-93a6-652ab4ec0624" (UID: "57757f1a-681c-436e-93a6-652ab4ec0624"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:24:25.350538 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.350407    5639 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-tmp-dir\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:25.350627 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.350551    5639 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-model-cache\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:25.350627 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.350565    5639 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-home\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:25.352087 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.352066    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-dshm" pod "57757f1a-681c-436e-93a6-652ab4ec0624" (UID: "57757f1a-681c-436e-93a6-652ab4ec0624"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:24:25.352171 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.352110    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57757f1a-681c-436e-93a6-652ab4ec0624-tls-certs" pod "57757f1a-681c-436e-93a6-652ab4ec0624" (UID: "57757f1a-681c-436e-93a6-652ab4ec0624"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:24:25.352246 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.352223    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57757f1a-681c-436e-93a6-652ab4ec0624-kube-api-access-l7s6g" pod "57757f1a-681c-436e-93a6-652ab4ec0624" (UID: "57757f1a-681c-436e-93a6-652ab4ec0624"). InnerVolumeSpecName "kube-api-access-l7s6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:24:25.451693 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.451662    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7s6g\" (UniqueName: \"kubernetes.io/projected/57757f1a-681c-436e-93a6-652ab4ec0624-kube-api-access-l7s6g\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:25.451693 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.451690    5639 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57757f1a-681c-436e-93a6-652ab4ec0624-dshm\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:25.451693 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.451700    5639 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57757f1a-681c-436e-93a6-652ab4ec0624-tls-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:25.639901 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.639863    5639 generic.go:356] "Generic (PLEG): container finished" podID="57757f1a-681c-436e-93a6-652ab4ec0624" containerID="6edd2b954bf13e6ab653b2a9ace90dd4ee3fb96fd55b3ca2e02a15c6a4a03c07" exitCode=0
Nov 02 02:24:25.640120 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.639919    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"
Nov 02 02:24:25.640120 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.639941    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w" event={"ID":"57757f1a-681c-436e-93a6-652ab4ec0624","Type":"ContainerDied","Data":"6edd2b954bf13e6ab653b2a9ace90dd4ee3fb96fd55b3ca2e02a15c6a4a03c07"}
Nov 02 02:24:25.640120 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.639991    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w" event={"ID":"57757f1a-681c-436e-93a6-652ab4ec0624","Type":"ContainerDied","Data":"13f9d2ef72e09fbdc621a6fda42f069d42a39cd26711e211c758769f807c6f54"}
Nov 02 02:24:25.640120 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.640011    5639 scope.go:122] "RemoveContainer" containerID="6edd2b954bf13e6ab653b2a9ace90dd4ee3fb96fd55b3ca2e02a15c6a4a03c07"
Nov 02 02:24:25.650216 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.650191    5639 scope.go:122] "RemoveContainer" containerID="6edd2b954bf13e6ab653b2a9ace90dd4ee3fb96fd55b3ca2e02a15c6a4a03c07"
Nov 02 02:24:25.650580 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:24:25.650555    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6edd2b954bf13e6ab653b2a9ace90dd4ee3fb96fd55b3ca2e02a15c6a4a03c07\": container with ID starting with 6edd2b954bf13e6ab653b2a9ace90dd4ee3fb96fd55b3ca2e02a15c6a4a03c07 not found: ID does not exist" containerID="6edd2b954bf13e6ab653b2a9ace90dd4ee3fb96fd55b3ca2e02a15c6a4a03c07"
Nov 02 02:24:25.650667 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.650595    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6edd2b954bf13e6ab653b2a9ace90dd4ee3fb96fd55b3ca2e02a15c6a4a03c07"} err="failed to get container status \"6edd2b954bf13e6ab653b2a9ace90dd4ee3fb96fd55b3ca2e02a15c6a4a03c07\": rpc error: code = NotFound desc = could not find container \"6edd2b954bf13e6ab653b2a9ace90dd4ee3fb96fd55b3ca2e02a15c6a4a03c07\": container with ID starting with 6edd2b954bf13e6ab653b2a9ace90dd4ee3fb96fd55b3ca2e02a15c6a4a03c07 not found: ID does not exist"
Nov 02 02:24:25.803847 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.803813    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"]
Nov 02 02:24:25.808141 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:25.808115    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-fa3747/simulator-single-main-fa3747-kserve-7784c8c6b4-z6d7w"]
Nov 02 02:24:27.217678 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:27.217644    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57757f1a-681c-436e-93a6-652ab4ec0624" path="/var/lib/kubelet/pods/57757f1a-681c-436e-93a6-652ab4ec0624/volumes"
Nov 02 02:24:27.271550 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:27.271513    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:24:27.443683 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:24:27.443638    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:24:31.164429 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:31.164396    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"]
Nov 02 02:24:31.164948 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:31.164598    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq" podUID="9e865a45-c3b0-4163-bf86-73f839f3cb8d" containerName="istio-proxy" containerID="cri-o://81d117c1b322d3f8a82f38ddbe8738fffed890f95174fbb3fc385e0e90299658" gracePeriod=30
Nov 02 02:24:32.722229 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:32.721795    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nprts"]
Nov 02 02:24:32.769080 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:32.769032    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nprts"]
Nov 02 02:24:32.841907 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:32.841866    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9swvr"]
Nov 02 02:24:32.842466 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:32.842444    5639 cpu_manager.go:414] "containerMap: removing container" podUID="8a703dd1-e3c2-498a-b25e-6146c8192168" containerName="revoke-keys"
Nov 02 02:24:32.842466 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:32.842466    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a703dd1-e3c2-498a-b25e-6146c8192168" containerName="revoke-keys"
Nov 02 02:24:32.842636 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:32.842490    5639 cpu_manager.go:414] "containerMap: removing container" podUID="57757f1a-681c-436e-93a6-652ab4ec0624" containerName="main"
Nov 02 02:24:32.842636 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:32.842500    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57757f1a-681c-436e-93a6-652ab4ec0624" containerName="main"
Nov 02 02:24:32.842636 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:32.842583    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="57757f1a-681c-436e-93a6-652ab4ec0624" containerName="main"
Nov 02 02:24:32.842636 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:32.842598    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a703dd1-e3c2-498a-b25e-6146c8192168" containerName="revoke-keys"
Nov 02 02:24:32.847240 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:32.847218    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9swvr"
Nov 02 02:24:32.900523 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:32.900487    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:24:32.901006 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:32.900541    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:24:32.901006 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:32.900568    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:24:32.901006 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:32.900623    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:24:32.902861 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:32.902837    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9swvr"]
Nov 02 02:24:32.937126 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:32.937088    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8866053-37ae-4643-a307-95b0542ac721-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9swvr\" (UID: \"e8866053-37ae-4643-a307-95b0542ac721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9swvr"
Nov 02 02:24:32.937345 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:32.937222    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mghw4\" (UniqueName: \"kubernetes.io/projected/e8866053-37ae-4643-a307-95b0542ac721-kube-api-access-mghw4\") pod \"maas-api-revoke-sub-3d4156f1-9swvr\" (UID: \"e8866053-37ae-4643-a307-95b0542ac721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9swvr"
Nov 02 02:24:32.937345 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:32.937319    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8866053-37ae-4643-a307-95b0542ac721-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9swvr\" (UID: \"e8866053-37ae-4643-a307-95b0542ac721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9swvr"
Nov 02 02:24:33.038520 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:33.038424    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8866053-37ae-4643-a307-95b0542ac721-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9swvr\" (UID: \"e8866053-37ae-4643-a307-95b0542ac721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9swvr"
Nov 02 02:24:33.038520 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:33.038492    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8866053-37ae-4643-a307-95b0542ac721-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9swvr\" (UID: \"e8866053-37ae-4643-a307-95b0542ac721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9swvr"
Nov 02 02:24:33.038752 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:33.038541    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mghw4\" (UniqueName: \"kubernetes.io/projected/e8866053-37ae-4643-a307-95b0542ac721-kube-api-access-mghw4\") pod \"maas-api-revoke-sub-3d4156f1-9swvr\" (UID: \"e8866053-37ae-4643-a307-95b0542ac721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9swvr"
Nov 02 02:24:33.039152 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:33.039129    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8866053-37ae-4643-a307-95b0542ac721-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9swvr\" (UID: \"e8866053-37ae-4643-a307-95b0542ac721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9swvr"
Nov 02 02:24:33.040974 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:33.040935    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8866053-37ae-4643-a307-95b0542ac721-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9swvr\" (UID: \"e8866053-37ae-4643-a307-95b0542ac721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9swvr"
Nov 02 02:24:33.099006 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:33.098970    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mghw4\" (UniqueName: \"kubernetes.io/projected/e8866053-37ae-4643-a307-95b0542ac721-kube-api-access-mghw4\") pod \"maas-api-revoke-sub-3d4156f1-9swvr\" (UID: \"e8866053-37ae-4643-a307-95b0542ac721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9swvr"
Nov 02 02:24:33.160817 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:33.160781    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9swvr"
Nov 02 02:24:33.220199 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:33.220166    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd9e71a7-b87a-4a05-8857-9dc35de0a064" path="/var/lib/kubelet/pods/cd9e71a7-b87a-4a05-8857-9dc35de0a064/volumes"
Nov 02 02:24:33.540323 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:33.540141    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9swvr"]
Nov 02 02:24:33.543105 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:24:33.543077    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8866053_37ae_4643_a307_95b0542ac721.slice/crio-1db0ce5fcb0bafe477cb1f8008d28de8d55fb6ae10b27b66efa898be5873209c WatchSource:0}: Error finding container 1db0ce5fcb0bafe477cb1f8008d28de8d55fb6ae10b27b66efa898be5873209c: Status 404 returned error can't find the container with id 1db0ce5fcb0bafe477cb1f8008d28de8d55fb6ae10b27b66efa898be5873209c
Nov 02 02:24:33.688033 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:33.687990    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9swvr" event={"ID":"e8866053-37ae-4643-a307-95b0542ac721","Type":"ContainerStarted","Data":"1db0ce5fcb0bafe477cb1f8008d28de8d55fb6ae10b27b66efa898be5873209c"}
Nov 02 02:24:34.695783 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:34.695741    5639 generic.go:356] "Generic (PLEG): container finished" podID="e8866053-37ae-4643-a307-95b0542ac721" containerID="708910316a1f666a162d13f9e5b7c298076ffcd5e61fb6de50db9179fb2b2ff5" exitCode=0
Nov 02 02:24:34.696198 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:34.695818    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9swvr" event={"ID":"e8866053-37ae-4643-a307-95b0542ac721","Type":"ContainerDied","Data":"708910316a1f666a162d13f9e5b7c298076ffcd5e61fb6de50db9179fb2b2ff5"}
Nov 02 02:24:35.091525 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:35.091485    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf" podUID="00036f2e-46c1-46c9-b9b6-d987a40f1c77" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:24:35.091697 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:35.091485    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp" podUID="a664188c-3afb-41b4-a941-ae65055f8874" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:24:35.849795 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:35.849760    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9swvr"
Nov 02 02:24:35.866158 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:35.866126    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8866053-37ae-4643-a307-95b0542ac721-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8866053-37ae-4643-a307-95b0542ac721-maas-api-openshift-ca\") pod \"e8866053-37ae-4643-a307-95b0542ac721\" (UID: \"e8866053-37ae-4643-a307-95b0542ac721\") "
Nov 02 02:24:35.866357 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:35.866249    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8866053-37ae-4643-a307-95b0542ac721-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8866053-37ae-4643-a307-95b0542ac721-maas-api-xks-ca\") pod \"e8866053-37ae-4643-a307-95b0542ac721\" (UID: \"e8866053-37ae-4643-a307-95b0542ac721\") "
Nov 02 02:24:35.866415 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:35.866381    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8866053-37ae-4643-a307-95b0542ac721-kube-api-access-mghw4\" (UniqueName: \"kubernetes.io/projected/e8866053-37ae-4643-a307-95b0542ac721-kube-api-access-mghw4\") pod \"e8866053-37ae-4643-a307-95b0542ac721\" (UID: \"e8866053-37ae-4643-a307-95b0542ac721\") "
Nov 02 02:24:35.866526 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:35.866503    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8866053-37ae-4643-a307-95b0542ac721-maas-api-openshift-ca" pod "e8866053-37ae-4643-a307-95b0542ac721" (UID: "e8866053-37ae-4643-a307-95b0542ac721"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:24:35.866702 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:35.866688    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8866053-37ae-4643-a307-95b0542ac721-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:35.868572 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:35.868549    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8866053-37ae-4643-a307-95b0542ac721-maas-api-xks-ca" pod "e8866053-37ae-4643-a307-95b0542ac721" (UID: "e8866053-37ae-4643-a307-95b0542ac721"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:24:35.868731 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:35.868706    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8866053-37ae-4643-a307-95b0542ac721-kube-api-access-mghw4" pod "e8866053-37ae-4643-a307-95b0542ac721" (UID: "e8866053-37ae-4643-a307-95b0542ac721"). InnerVolumeSpecName "kube-api-access-mghw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:24:35.967762 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:35.967667    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mghw4\" (UniqueName: \"kubernetes.io/projected/e8866053-37ae-4643-a307-95b0542ac721-kube-api-access-mghw4\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:35.967762 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:35.967695    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8866053-37ae-4643-a307-95b0542ac721-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:36.412298 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.412258    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:24:36.472954 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.472917    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-envoy\") pod \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") "
Nov 02 02:24:36.473165 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.473034    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-workload-socket\") pod \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") "
Nov 02 02:24:36.473165 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.473056    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-data\") pod \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") "
Nov 02 02:24:36.473165 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.473103    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-token\" (UniqueName: \"kubernetes.io/projected/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-token\") pod \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") "
Nov 02 02:24:36.473165 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.473128    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istiod-ca-cert\") pod \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") "
Nov 02 02:24:36.473165 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.473162    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-credential-socket\") pod \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") "
Nov 02 02:24:36.473489 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.473190    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-workload-certs\") pod \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") "
Nov 02 02:24:36.473489 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.473232    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-podinfo\") pod \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") "
Nov 02 02:24:36.473489 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.473410    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e865a45-c3b0-4163-bf86-73f839f3cb8d-kube-api-access-blz2s\" (UniqueName: \"kubernetes.io/projected/9e865a45-c3b0-4163-bf86-73f839f3cb8d-kube-api-access-blz2s\") pod \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\" (UID: \"9e865a45-c3b0-4163-bf86-73f839f3cb8d\") "
Nov 02 02:24:36.473489 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.473411    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-workload-socket" pod "9e865a45-c3b0-4163-bf86-73f839f3cb8d" (UID: "9e865a45-c3b0-4163-bf86-73f839f3cb8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:24:36.473489 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.473420    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-data" pod "9e865a45-c3b0-4163-bf86-73f839f3cb8d" (UID: "9e865a45-c3b0-4163-bf86-73f839f3cb8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:24:36.473741 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.473510    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-credential-socket" pod "9e865a45-c3b0-4163-bf86-73f839f3cb8d" (UID: "9e865a45-c3b0-4163-bf86-73f839f3cb8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:24:36.473741 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.473614    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istiod-ca-cert" pod "9e865a45-c3b0-4163-bf86-73f839f3cb8d" (UID: "9e865a45-c3b0-4163-bf86-73f839f3cb8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:24:36.473741 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.473702    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-workload-certs" pod "9e865a45-c3b0-4163-bf86-73f839f3cb8d" (UID: "9e865a45-c3b0-4163-bf86-73f839f3cb8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:24:36.473952 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.473869    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:36.473952 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.473889    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:36.473952 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.473904    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:36.473952 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.473917    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:36.473952 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.473932    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:36.475627 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.475604    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-token" pod "9e865a45-c3b0-4163-bf86-73f839f3cb8d" (UID: "9e865a45-c3b0-4163-bf86-73f839f3cb8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:24:36.475757 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.475626    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-envoy" pod "9e865a45-c3b0-4163-bf86-73f839f3cb8d" (UID: "9e865a45-c3b0-4163-bf86-73f839f3cb8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:24:36.475757 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.475625    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-podinfo" pod "9e865a45-c3b0-4163-bf86-73f839f3cb8d" (UID: "9e865a45-c3b0-4163-bf86-73f839f3cb8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:24:36.475757 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.475678    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e865a45-c3b0-4163-bf86-73f839f3cb8d-kube-api-access-blz2s" pod "9e865a45-c3b0-4163-bf86-73f839f3cb8d" (UID: "9e865a45-c3b0-4163-bf86-73f839f3cb8d"). InnerVolumeSpecName "kube-api-access-blz2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:24:36.575122 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.575039    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:36.575122 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.575069    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blz2s\" (UniqueName: \"kubernetes.io/projected/9e865a45-c3b0-4163-bf86-73f839f3cb8d-kube-api-access-blz2s\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:36.575122 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.575079    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:36.575122 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.575088    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e865a45-c3b0-4163-bf86-73f839f3cb8d-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:36.708442 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.708407    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9swvr"
Nov 02 02:24:36.708669 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.708407    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9swvr" event={"ID":"e8866053-37ae-4643-a307-95b0542ac721","Type":"ContainerDied","Data":"1db0ce5fcb0bafe477cb1f8008d28de8d55fb6ae10b27b66efa898be5873209c"}
Nov 02 02:24:36.708669 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.708532    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1db0ce5fcb0bafe477cb1f8008d28de8d55fb6ae10b27b66efa898be5873209c"
Nov 02 02:24:36.710358 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.710317    5639 generic.go:356] "Generic (PLEG): container finished" podID="9e865a45-c3b0-4163-bf86-73f839f3cb8d" containerID="81d117c1b322d3f8a82f38ddbe8738fffed890f95174fbb3fc385e0e90299658" exitCode=0
Nov 02 02:24:36.710497 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.710360    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq" event={"ID":"9e865a45-c3b0-4163-bf86-73f839f3cb8d","Type":"ContainerDied","Data":"81d117c1b322d3f8a82f38ddbe8738fffed890f95174fbb3fc385e0e90299658"}
Nov 02 02:24:36.710497 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.710374    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"
Nov 02 02:24:36.710497 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.710414    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq" event={"ID":"9e865a45-c3b0-4163-bf86-73f839f3cb8d","Type":"ContainerDied","Data":"bc4d59b9ee96e7f2cd4c58342c9c54c85ca96d80306abac22f56191004a94bc4"}
Nov 02 02:24:36.710497 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.710430    5639 scope.go:122] "RemoveContainer" containerID="81d117c1b322d3f8a82f38ddbe8738fffed890f95174fbb3fc385e0e90299658"
Nov 02 02:24:36.728344 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.728320    5639 scope.go:122] "RemoveContainer" containerID="37f3ad868bf56a1dc64821863283742e014dbb0dae5e790508e70825d228d306"
Nov 02 02:24:36.738008 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.737980    5639 scope.go:122] "RemoveContainer" containerID="81d117c1b322d3f8a82f38ddbe8738fffed890f95174fbb3fc385e0e90299658"
Nov 02 02:24:36.738416 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:24:36.738390    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81d117c1b322d3f8a82f38ddbe8738fffed890f95174fbb3fc385e0e90299658\": container with ID starting with 81d117c1b322d3f8a82f38ddbe8738fffed890f95174fbb3fc385e0e90299658 not found: ID does not exist" containerID="81d117c1b322d3f8a82f38ddbe8738fffed890f95174fbb3fc385e0e90299658"
Nov 02 02:24:36.738503 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.738425    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d117c1b322d3f8a82f38ddbe8738fffed890f95174fbb3fc385e0e90299658"} err="failed to get container status \"81d117c1b322d3f8a82f38ddbe8738fffed890f95174fbb3fc385e0e90299658\": rpc error: code = NotFound desc = could not find container \"81d117c1b322d3f8a82f38ddbe8738fffed890f95174fbb3fc385e0e90299658\": container with ID starting with 81d117c1b322d3f8a82f38ddbe8738fffed890f95174fbb3fc385e0e90299658 not found: ID does not exist"
Nov 02 02:24:36.738503 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.738449    5639 scope.go:122] "RemoveContainer" containerID="37f3ad868bf56a1dc64821863283742e014dbb0dae5e790508e70825d228d306"
Nov 02 02:24:36.738708 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:24:36.738689    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37f3ad868bf56a1dc64821863283742e014dbb0dae5e790508e70825d228d306\": container with ID starting with 37f3ad868bf56a1dc64821863283742e014dbb0dae5e790508e70825d228d306 not found: ID does not exist" containerID="37f3ad868bf56a1dc64821863283742e014dbb0dae5e790508e70825d228d306"
Nov 02 02:24:36.738746 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.738715    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37f3ad868bf56a1dc64821863283742e014dbb0dae5e790508e70825d228d306"} err="failed to get container status \"37f3ad868bf56a1dc64821863283742e014dbb0dae5e790508e70825d228d306\": rpc error: code = NotFound desc = could not find container \"37f3ad868bf56a1dc64821863283742e014dbb0dae5e790508e70825d228d306\": container with ID starting with 37f3ad868bf56a1dc64821863283742e014dbb0dae5e790508e70825d228d306 not found: ID does not exist"
Nov 02 02:24:36.876209 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.876176    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"]
Nov 02 02:24:36.880206 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:36.880178    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fa3747-openshift-default-6c567d475cczkcq"]
Nov 02 02:24:37.221672 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:37.221639    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e865a45-c3b0-4163-bf86-73f839f3cb8d" path="/var/lib/kubelet/pods/9e865a45-c3b0-4163-bf86-73f839f3cb8d/volumes"
Nov 02 02:24:39.047814 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:39.047774    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lqjpk"]
Nov 02 02:24:39.097575 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:39.097536    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lqjpk"]
Nov 02 02:24:39.217794 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:39.217762    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08ef4ca8-74b3-4d03-b24d-2d421d42edf4" path="/var/lib/kubelet/pods/08ef4ca8-74b3-4d03-b24d-2d421d42edf4/volumes"
Nov 02 02:24:39.540045 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:39.540002    5639 scope.go:122] "RemoveContainer" containerID="d5aaaa83ba7c735b1ac0efcc02db9778ec7d36e7bc02e3d4367f79c6f6dc879b"
Nov 02 02:24:39.549682 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:39.549660    5639 scope.go:122] "RemoveContainer" containerID="8150bba3eb1322a04d9b4a160de7ffff45b88491e836d6928040967cc5167f34"
Nov 02 02:24:39.558922 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:39.558895    5639 scope.go:122] "RemoveContainer" containerID="9f66d2b9a3a6a45f2cb79206b2976b0427677df60dd69c0cf9b907da71310c3c"
Nov 02 02:24:42.317112 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.317043    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf"
Nov 02 02:24:42.401767 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.401740    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp"
Nov 02 02:24:42.434595 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.434504    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00036f2e-46c1-46c9-b9b6-d987a40f1c77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00036f2e-46c1-46c9-b9b6-d987a40f1c77-plugins-config-volume\") pod \"00036f2e-46c1-46c9-b9b6-d987a40f1c77\" (UID: \"00036f2e-46c1-46c9-b9b6-d987a40f1c77\") "
Nov 02 02:24:42.434595 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.434578    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00036f2e-46c1-46c9-b9b6-d987a40f1c77-kube-api-access-bxt7g\" (UniqueName: \"kubernetes.io/projected/00036f2e-46c1-46c9-b9b6-d987a40f1c77-kube-api-access-bxt7g\") pod \"00036f2e-46c1-46c9-b9b6-d987a40f1c77\" (UID: \"00036f2e-46c1-46c9-b9b6-d987a40f1c77\") "
Nov 02 02:24:42.435194 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.434916    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00036f2e-46c1-46c9-b9b6-d987a40f1c77-plugins-config-volume" pod "00036f2e-46c1-46c9-b9b6-d987a40f1c77" (UID: "00036f2e-46c1-46c9-b9b6-d987a40f1c77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:24:42.435194 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.435000    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00036f2e-46c1-46c9-b9b6-d987a40f1c77-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:42.437637 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.437609    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00036f2e-46c1-46c9-b9b6-d987a40f1c77-kube-api-access-bxt7g" pod "00036f2e-46c1-46c9-b9b6-d987a40f1c77" (UID: "00036f2e-46c1-46c9-b9b6-d987a40f1c77"). InnerVolumeSpecName "kube-api-access-bxt7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:24:42.535541 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.535507    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a664188c-3afb-41b4-a941-ae65055f8874-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a664188c-3afb-41b4-a941-ae65055f8874-plugins-config-volume\") pod \"a664188c-3afb-41b4-a941-ae65055f8874\" (UID: \"a664188c-3afb-41b4-a941-ae65055f8874\") "
Nov 02 02:24:42.535738 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.535566    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a664188c-3afb-41b4-a941-ae65055f8874-kube-api-access-x24n9\" (UniqueName: \"kubernetes.io/projected/a664188c-3afb-41b4-a941-ae65055f8874-kube-api-access-x24n9\") pod \"a664188c-3afb-41b4-a941-ae65055f8874\" (UID: \"a664188c-3afb-41b4-a941-ae65055f8874\") "
Nov 02 02:24:42.535872 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.535860    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxt7g\" (UniqueName: \"kubernetes.io/projected/00036f2e-46c1-46c9-b9b6-d987a40f1c77-kube-api-access-bxt7g\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:42.536043 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.536009    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a664188c-3afb-41b4-a941-ae65055f8874-plugins-config-volume" pod "a664188c-3afb-41b4-a941-ae65055f8874" (UID: "a664188c-3afb-41b4-a941-ae65055f8874"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:24:42.537953 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.537921    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a664188c-3afb-41b4-a941-ae65055f8874-kube-api-access-x24n9" pod "a664188c-3afb-41b4-a941-ae65055f8874" (UID: "a664188c-3afb-41b4-a941-ae65055f8874"). InnerVolumeSpecName "kube-api-access-x24n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:24:42.637219 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.637187    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a664188c-3afb-41b4-a941-ae65055f8874-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:42.637219 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.637217    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x24n9\" (UniqueName: \"kubernetes.io/projected/a664188c-3afb-41b4-a941-ae65055f8874-kube-api-access-x24n9\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:42.743864 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.743830    5639 generic.go:356] "Generic (PLEG): container finished" podID="a664188c-3afb-41b4-a941-ae65055f8874" containerID="4099ecb2abec0d4f35afd5702482cabdd68f6c8a035c7f996504bad2eb2bbcd8" exitCode=0
Nov 02 02:24:42.744078 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.743908    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp"
Nov 02 02:24:42.744078 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.743928    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp" event={"ID":"a664188c-3afb-41b4-a941-ae65055f8874","Type":"ContainerDied","Data":"4099ecb2abec0d4f35afd5702482cabdd68f6c8a035c7f996504bad2eb2bbcd8"}
Nov 02 02:24:42.744078 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.744018    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp" event={"ID":"a664188c-3afb-41b4-a941-ae65055f8874","Type":"ContainerDied","Data":"609e36cd685ed4264a81acff3de8aef5b1854fb9b6f2baf96028a14afad1ae20"}
Nov 02 02:24:42.744078 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.744056    5639 scope.go:122] "RemoveContainer" containerID="4099ecb2abec0d4f35afd5702482cabdd68f6c8a035c7f996504bad2eb2bbcd8"
Nov 02 02:24:42.745599 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.745573    5639 generic.go:356] "Generic (PLEG): container finished" podID="00036f2e-46c1-46c9-b9b6-d987a40f1c77" containerID="bed339ff41457fb83e0b104b9147bf2caaa37973dac585c252f5255315c6fc7e" exitCode=0
Nov 02 02:24:42.745894 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.745631    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf"
Nov 02 02:24:42.745894 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.745657    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf" event={"ID":"00036f2e-46c1-46c9-b9b6-d987a40f1c77","Type":"ContainerDied","Data":"bed339ff41457fb83e0b104b9147bf2caaa37973dac585c252f5255315c6fc7e"}
Nov 02 02:24:42.745894 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.745724    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf" event={"ID":"00036f2e-46c1-46c9-b9b6-d987a40f1c77","Type":"ContainerDied","Data":"9c19d026c714b303a94ff9d0053772978b0eb91ac5f1f59c1dc7188a2f865ec9"}
Nov 02 02:24:42.755639 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.755605    5639 scope.go:122] "RemoveContainer" containerID="4099ecb2abec0d4f35afd5702482cabdd68f6c8a035c7f996504bad2eb2bbcd8"
Nov 02 02:24:42.756175 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:24:42.756143    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4099ecb2abec0d4f35afd5702482cabdd68f6c8a035c7f996504bad2eb2bbcd8\": container with ID starting with 4099ecb2abec0d4f35afd5702482cabdd68f6c8a035c7f996504bad2eb2bbcd8 not found: ID does not exist" containerID="4099ecb2abec0d4f35afd5702482cabdd68f6c8a035c7f996504bad2eb2bbcd8"
Nov 02 02:24:42.756344 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.756189    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4099ecb2abec0d4f35afd5702482cabdd68f6c8a035c7f996504bad2eb2bbcd8"} err="failed to get container status \"4099ecb2abec0d4f35afd5702482cabdd68f6c8a035c7f996504bad2eb2bbcd8\": rpc error: code = NotFound desc = could not find container \"4099ecb2abec0d4f35afd5702482cabdd68f6c8a035c7f996504bad2eb2bbcd8\": container with ID starting with 4099ecb2abec0d4f35afd5702482cabdd68f6c8a035c7f996504bad2eb2bbcd8 not found: ID does not exist"
Nov 02 02:24:42.756344 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.756215    5639 scope.go:122] "RemoveContainer" containerID="bed339ff41457fb83e0b104b9147bf2caaa37973dac585c252f5255315c6fc7e"
Nov 02 02:24:42.765330 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.765312    5639 scope.go:122] "RemoveContainer" containerID="bed339ff41457fb83e0b104b9147bf2caaa37973dac585c252f5255315c6fc7e"
Nov 02 02:24:42.765614 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:24:42.765590    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bed339ff41457fb83e0b104b9147bf2caaa37973dac585c252f5255315c6fc7e\": container with ID starting with bed339ff41457fb83e0b104b9147bf2caaa37973dac585c252f5255315c6fc7e not found: ID does not exist" containerID="bed339ff41457fb83e0b104b9147bf2caaa37973dac585c252f5255315c6fc7e"
Nov 02 02:24:42.765679 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.765630    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bed339ff41457fb83e0b104b9147bf2caaa37973dac585c252f5255315c6fc7e"} err="failed to get container status \"bed339ff41457fb83e0b104b9147bf2caaa37973dac585c252f5255315c6fc7e\": rpc error: code = NotFound desc = could not find container \"bed339ff41457fb83e0b104b9147bf2caaa37973dac585c252f5255315c6fc7e\": container with ID starting with bed339ff41457fb83e0b104b9147bf2caaa37973dac585c252f5255315c6fc7e not found: ID does not exist"
Nov 02 02:24:42.909015 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.908925    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp"]
Nov 02 02:24:42.912029 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:42.912004    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa3747-cb554tsfwp"]
Nov 02 02:24:43.020824 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:43.020782    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf"]
Nov 02 02:24:43.024338 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:43.024308    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fa3747-d48d667-h6xnf"]
Nov 02 02:24:43.218664 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:43.218579    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00036f2e-46c1-46c9-b9b6-d987a40f1c77" path="/var/lib/kubelet/pods/00036f2e-46c1-46c9-b9b6-d987a40f1c77/volumes"
Nov 02 02:24:43.219075 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:43.219054    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a664188c-3afb-41b4-a941-ae65055f8874" path="/var/lib/kubelet/pods/a664188c-3afb-41b4-a941-ae65055f8874/volumes"
Nov 02 02:24:49.278862 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:49.278825    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:24:57.420254 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:24:57.420215    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:24:59.779856 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:24:59.779820    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:25:03.046828 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:25:03.046790    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fcbfq"]
Nov 02 02:25:03.096366 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:25:03.096328    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fcbfq"]
Nov 02 02:25:03.218133 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:25:03.218101    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8ec2711-d219-47b2-90b6-13310e016f3f" path="/var/lib/kubelet/pods/e8ec2711-d219-47b2-90b6-13310e016f3f/volumes"
Nov 02 02:25:27.421420 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:25:27.421381    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:25:39.600126 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:25:39.600088    5639 scope.go:122] "RemoveContainer" containerID="e037fa688ced19da343b59ca11dbc07a611697fd6b9befbc4d4df61682c58220"
Nov 02 02:25:39.609927 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:25:39.609904    5639 scope.go:122] "RemoveContainer" containerID="99b29efa271caadfc9b614c5e8ed8b60ad3a5c25490b5b57d5c5b7a736180e87"
Nov 02 02:25:57.422736 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:25:57.422685    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:26:27.422083 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:26:27.422044    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:26:30.718824 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:30.718783    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:26:35.290407 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.290373    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d55g7"]
Nov 02 02:26:35.290857 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.290780    5639 cpu_manager.go:414] "containerMap: removing container" podUID="9e865a45-c3b0-4163-bf86-73f839f3cb8d" containerName="istio-proxy"
Nov 02 02:26:35.290857 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.290791    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e865a45-c3b0-4163-bf86-73f839f3cb8d" containerName="istio-proxy"
Nov 02 02:26:35.290857 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.290804    5639 cpu_manager.go:414] "containerMap: removing container" podUID="e8866053-37ae-4643-a307-95b0542ac721" containerName="revoke-keys"
Nov 02 02:26:35.290857 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.290809    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8866053-37ae-4643-a307-95b0542ac721" containerName="revoke-keys"
Nov 02 02:26:35.290857 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.290830    5639 cpu_manager.go:414] "containerMap: removing container" podUID="a664188c-3afb-41b4-a941-ae65055f8874" containerName="payload-pre-processing"
Nov 02 02:26:35.290857 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.290836    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a664188c-3afb-41b4-a941-ae65055f8874" containerName="payload-pre-processing"
Nov 02 02:26:35.290857 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.290849    5639 cpu_manager.go:414] "containerMap: removing container" podUID="00036f2e-46c1-46c9-b9b6-d987a40f1c77" containerName="payload-processing"
Nov 02 02:26:35.290857 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.290854    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00036f2e-46c1-46c9-b9b6-d987a40f1c77" containerName="payload-processing"
Nov 02 02:26:35.291126 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.290923    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="a664188c-3afb-41b4-a941-ae65055f8874" containerName="payload-pre-processing"
Nov 02 02:26:35.291126 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.290932    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e865a45-c3b0-4163-bf86-73f839f3cb8d" containerName="istio-proxy"
Nov 02 02:26:35.291126 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.290937    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="00036f2e-46c1-46c9-b9b6-d987a40f1c77" containerName="payload-processing"
Nov 02 02:26:35.291126 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.290948    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e865a45-c3b0-4163-bf86-73f839f3cb8d" containerName="istio-proxy"
Nov 02 02:26:35.291126 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.290955    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8866053-37ae-4643-a307-95b0542ac721" containerName="revoke-keys"
Nov 02 02:26:35.302465 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.302432    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d55g7"
Nov 02 02:26:35.351628 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.351599    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d55g7"]
Nov 02 02:26:35.375771 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.375728    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e46eb581-da6b-46fe-beae-821c3b17bcd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-d55g7\" (UID: \"e46eb581-da6b-46fe-beae-821c3b17bcd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d55g7"
Nov 02 02:26:35.375981 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.375898    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjxn6\" (UniqueName: \"kubernetes.io/projected/e46eb581-da6b-46fe-beae-821c3b17bcd2-kube-api-access-kjxn6\") pod \"maas-api-revoke-sub-d576e68b-d55g7\" (UID: \"e46eb581-da6b-46fe-beae-821c3b17bcd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d55g7"
Nov 02 02:26:35.375981 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.375935    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e46eb581-da6b-46fe-beae-821c3b17bcd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-d55g7\" (UID: \"e46eb581-da6b-46fe-beae-821c3b17bcd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d55g7"
Nov 02 02:26:35.399367 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.399337    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:26:35.399367 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.399346    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:26:35.399367 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.399369    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:26:35.404125 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.404106    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:26:35.477507 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.477471    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjxn6\" (UniqueName: \"kubernetes.io/projected/e46eb581-da6b-46fe-beae-821c3b17bcd2-kube-api-access-kjxn6\") pod \"maas-api-revoke-sub-d576e68b-d55g7\" (UID: \"e46eb581-da6b-46fe-beae-821c3b17bcd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d55g7"
Nov 02 02:26:35.477697 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.477524    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e46eb581-da6b-46fe-beae-821c3b17bcd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-d55g7\" (UID: \"e46eb581-da6b-46fe-beae-821c3b17bcd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d55g7"
Nov 02 02:26:35.477697 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.477564    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e46eb581-da6b-46fe-beae-821c3b17bcd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-d55g7\" (UID: \"e46eb581-da6b-46fe-beae-821c3b17bcd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d55g7"
Nov 02 02:26:35.478219 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.478200    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e46eb581-da6b-46fe-beae-821c3b17bcd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-d55g7\" (UID: \"e46eb581-da6b-46fe-beae-821c3b17bcd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d55g7"
Nov 02 02:26:35.480254 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.480231    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e46eb581-da6b-46fe-beae-821c3b17bcd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-d55g7\" (UID: \"e46eb581-da6b-46fe-beae-821c3b17bcd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d55g7"
Nov 02 02:26:35.538777 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.538745    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjxn6\" (UniqueName: \"kubernetes.io/projected/e46eb581-da6b-46fe-beae-821c3b17bcd2-kube-api-access-kjxn6\") pod \"maas-api-revoke-sub-d576e68b-d55g7\" (UID: \"e46eb581-da6b-46fe-beae-821c3b17bcd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d55g7"
Nov 02 02:26:35.615227 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.615195    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d55g7"
Nov 02 02:26:35.992258 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:35.992232    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d55g7"]
Nov 02 02:26:35.994682 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:26:35.994653    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode46eb581_da6b_46fe_beae_821c3b17bcd2.slice/crio-7e68fbc83341c1befc41d559acc0e0cd5d233cc30091552a72b06a53c56537cc WatchSource:0}: Error finding container 7e68fbc83341c1befc41d559acc0e0cd5d233cc30091552a72b06a53c56537cc: Status 404 returned error can't find the container with id 7e68fbc83341c1befc41d559acc0e0cd5d233cc30091552a72b06a53c56537cc
Nov 02 02:26:36.429529 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:36.429483    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d55g7" event={"ID":"e46eb581-da6b-46fe-beae-821c3b17bcd2","Type":"ContainerStarted","Data":"2f387a7ce05fddc603ed7bffe5d160654a6f337c1e7a4da7f8c603a51ec352be"}
Nov 02 02:26:36.430029 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:36.429546    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d55g7" event={"ID":"e46eb581-da6b-46fe-beae-821c3b17bcd2","Type":"ContainerStarted","Data":"7e68fbc83341c1befc41d559acc0e0cd5d233cc30091552a72b06a53c56537cc"}
Nov 02 02:26:36.546068 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:36.545949    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d55g7" podStartSLOduration=1.545933373 podStartE2EDuration="1.545933373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:26:36.545130872 +0000 UTC m=+3661.448812986" watchObservedRunningTime="2026-11-02 02:26:36.545933373 +0000 UTC m=+3661.449615508"
Nov 02 02:26:36.877578 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:26:36.877543    5639 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode46eb581_da6b_46fe_beae_821c3b17bcd2.slice/crio-conmon-2f387a7ce05fddc603ed7bffe5d160654a6f337c1e7a4da7f8c603a51ec352be.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 02:26:37.436819 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:37.436787    5639 generic.go:356] "Generic (PLEG): container finished" podID="e46eb581-da6b-46fe-beae-821c3b17bcd2" containerID="2f387a7ce05fddc603ed7bffe5d160654a6f337c1e7a4da7f8c603a51ec352be" exitCode=0
Nov 02 02:26:37.437257 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:37.436855    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d55g7" event={"ID":"e46eb581-da6b-46fe-beae-821c3b17bcd2","Type":"ContainerDied","Data":"2f387a7ce05fddc603ed7bffe5d160654a6f337c1e7a4da7f8c603a51ec352be"}
Nov 02 02:26:38.579709 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:38.579676    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d55g7"
Nov 02 02:26:38.711295 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:38.711183    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e46eb581-da6b-46fe-beae-821c3b17bcd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e46eb581-da6b-46fe-beae-821c3b17bcd2-maas-api-xks-ca\") pod \"e46eb581-da6b-46fe-beae-821c3b17bcd2\" (UID: \"e46eb581-da6b-46fe-beae-821c3b17bcd2\") "
Nov 02 02:26:38.711295 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:38.711234    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e46eb581-da6b-46fe-beae-821c3b17bcd2-kube-api-access-kjxn6\" (UniqueName: \"kubernetes.io/projected/e46eb581-da6b-46fe-beae-821c3b17bcd2-kube-api-access-kjxn6\") pod \"e46eb581-da6b-46fe-beae-821c3b17bcd2\" (UID: \"e46eb581-da6b-46fe-beae-821c3b17bcd2\") "
Nov 02 02:26:38.711574 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:38.711357    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e46eb581-da6b-46fe-beae-821c3b17bcd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e46eb581-da6b-46fe-beae-821c3b17bcd2-maas-api-openshift-ca\") pod \"e46eb581-da6b-46fe-beae-821c3b17bcd2\" (UID: \"e46eb581-da6b-46fe-beae-821c3b17bcd2\") "
Nov 02 02:26:38.711809 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:38.711778    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e46eb581-da6b-46fe-beae-821c3b17bcd2-maas-api-openshift-ca" pod "e46eb581-da6b-46fe-beae-821c3b17bcd2" (UID: "e46eb581-da6b-46fe-beae-821c3b17bcd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:26:38.714075 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:38.714043    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e46eb581-da6b-46fe-beae-821c3b17bcd2-maas-api-xks-ca" pod "e46eb581-da6b-46fe-beae-821c3b17bcd2" (UID: "e46eb581-da6b-46fe-beae-821c3b17bcd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:26:38.714201 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:38.714047    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e46eb581-da6b-46fe-beae-821c3b17bcd2-kube-api-access-kjxn6" pod "e46eb581-da6b-46fe-beae-821c3b17bcd2" (UID: "e46eb581-da6b-46fe-beae-821c3b17bcd2"). InnerVolumeSpecName "kube-api-access-kjxn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:26:38.813074 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:38.813034    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjxn6\" (UniqueName: \"kubernetes.io/projected/e46eb581-da6b-46fe-beae-821c3b17bcd2-kube-api-access-kjxn6\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:26:38.813074 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:38.813066    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e46eb581-da6b-46fe-beae-821c3b17bcd2-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:26:38.813074 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:38.813077    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e46eb581-da6b-46fe-beae-821c3b17bcd2-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:26:39.046710 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:39.046621    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-j8msj"]
Nov 02 02:26:39.049872 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:39.049843    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-j8msj"]
Nov 02 02:26:39.217609 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:39.217575    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea3a2c94-a15c-4214-8485-04316281816f" path="/var/lib/kubelet/pods/ea3a2c94-a15c-4214-8485-04316281816f/volumes"
Nov 02 02:26:39.451256 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:39.451219    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d55g7"
Nov 02 02:26:39.451488 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:39.451260    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d55g7" event={"ID":"e46eb581-da6b-46fe-beae-821c3b17bcd2","Type":"ContainerDied","Data":"7e68fbc83341c1befc41d559acc0e0cd5d233cc30091552a72b06a53c56537cc"}
Nov 02 02:26:39.451488 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:39.451343    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e68fbc83341c1befc41d559acc0e0cd5d233cc30091552a72b06a53c56537cc"
Nov 02 02:26:39.643890 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:39.643857    5639 scope.go:122] "RemoveContainer" containerID="c569fc17da2f2cae601872ee5b0709a9870f7bec9b243098582ed9dc1dfc9431"
Nov 02 02:26:39.652911 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:39.652887    5639 scope.go:122] "RemoveContainer" containerID="714557a30e85dd8550600552be5dfb59bdd458434af479ea73c9efdbf089e7bf"
Nov 02 02:26:48.108992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:48.108953    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:26:49.047409 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:49.047370    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-97pl6"]
Nov 02 02:26:49.071389 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:49.071355    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-97pl6"]
Nov 02 02:26:49.217714 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:49.217681    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="119c417e-f2c6-408b-9463-d451b1922653" path="/var/lib/kubelet/pods/119c417e-f2c6-408b-9463-d451b1922653/volumes"
Nov 02 02:26:55.371307 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:55.370421    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fd65x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:26:55.371307 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:55.370498    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x" podUID="faf2916d-803b-4f14-a2b1-1c45b742fc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:26:57.422039 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:26:57.421993    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:26:57.699543 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:26:57.699453    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:27:04.292063 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:04.292025    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bbmw9"]
Nov 02 02:27:04.292603 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:04.292474    5639 cpu_manager.go:414] "containerMap: removing container" podUID="e46eb581-da6b-46fe-beae-821c3b17bcd2" containerName="revoke-keys"
Nov 02 02:27:04.292603 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:04.292487    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e46eb581-da6b-46fe-beae-821c3b17bcd2" containerName="revoke-keys"
Nov 02 02:27:04.292603 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:04.292511    5639 cpu_manager.go:414] "containerMap: removing container" podUID="9e865a45-c3b0-4163-bf86-73f839f3cb8d" containerName="istio-proxy"
Nov 02 02:27:04.292603 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:04.292516    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e865a45-c3b0-4163-bf86-73f839f3cb8d" containerName="istio-proxy"
Nov 02 02:27:04.292603 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:04.292584    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="e46eb581-da6b-46fe-beae-821c3b17bcd2" containerName="revoke-keys"
Nov 02 02:27:04.297202 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:04.297180    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bbmw9"
Nov 02 02:27:04.335700 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:04.335658    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0de6c6-be7b-4863-8f63-b6ca94e7a6e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bbmw9\" (UID: \"df0de6c6-be7b-4863-8f63-b6ca94e7a6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bbmw9"
Nov 02 02:27:04.335895 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:04.335727    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0de6c6-be7b-4863-8f63-b6ca94e7a6e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bbmw9\" (UID: \"df0de6c6-be7b-4863-8f63-b6ca94e7a6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bbmw9"
Nov 02 02:27:04.335895 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:04.335828    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jchhz\" (UniqueName: \"kubernetes.io/projected/df0de6c6-be7b-4863-8f63-b6ca94e7a6e9-kube-api-access-jchhz\") pod \"maas-api-revoke-sub-d2bc93bc-bbmw9\" (UID: \"df0de6c6-be7b-4863-8f63-b6ca94e7a6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bbmw9"
Nov 02 02:27:04.351165 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:04.350936    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:27:04.351165 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:04.351097    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:27:04.400687 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:04.400649    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:27:04.400687 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:04.400651    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:27:04.402480 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:04.402456    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bbmw9"]
Nov 02 02:27:04.436634 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:04.436591    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jchhz\" (UniqueName: \"kubernetes.io/projected/df0de6c6-be7b-4863-8f63-b6ca94e7a6e9-kube-api-access-jchhz\") pod \"maas-api-revoke-sub-d2bc93bc-bbmw9\" (UID: \"df0de6c6-be7b-4863-8f63-b6ca94e7a6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bbmw9"
Nov 02 02:27:04.436833 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:04.436682    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0de6c6-be7b-4863-8f63-b6ca94e7a6e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bbmw9\" (UID: \"df0de6c6-be7b-4863-8f63-b6ca94e7a6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bbmw9"
Nov 02 02:27:04.436833 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:04.436724    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0de6c6-be7b-4863-8f63-b6ca94e7a6e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bbmw9\" (UID: \"df0de6c6-be7b-4863-8f63-b6ca94e7a6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bbmw9"
Nov 02 02:27:04.437424 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:04.437393    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0de6c6-be7b-4863-8f63-b6ca94e7a6e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bbmw9\" (UID: \"df0de6c6-be7b-4863-8f63-b6ca94e7a6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bbmw9"
Nov 02 02:27:04.439290 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:04.439250    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0de6c6-be7b-4863-8f63-b6ca94e7a6e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bbmw9\" (UID: \"df0de6c6-be7b-4863-8f63-b6ca94e7a6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bbmw9"
Nov 02 02:27:04.497680 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:04.497649    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jchhz\" (UniqueName: \"kubernetes.io/projected/df0de6c6-be7b-4863-8f63-b6ca94e7a6e9-kube-api-access-jchhz\") pod \"maas-api-revoke-sub-d2bc93bc-bbmw9\" (UID: \"df0de6c6-be7b-4863-8f63-b6ca94e7a6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bbmw9"
Nov 02 02:27:04.610439 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:04.610402    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bbmw9"
Nov 02 02:27:04.985260 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:04.985233    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bbmw9"]
Nov 02 02:27:04.988149 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:27:04.988110    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf0de6c6_be7b_4863_8f63_b6ca94e7a6e9.slice/crio-820a6362fc57881b4f8e0894337b04789cd118222efa1b3c679a59d5148bf9c5 WatchSource:0}: Error finding container 820a6362fc57881b4f8e0894337b04789cd118222efa1b3c679a59d5148bf9c5: Status 404 returned error can't find the container with id 820a6362fc57881b4f8e0894337b04789cd118222efa1b3c679a59d5148bf9c5
Nov 02 02:27:05.610733 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:05.610696    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bbmw9" event={"ID":"df0de6c6-be7b-4863-8f63-b6ca94e7a6e9","Type":"ContainerStarted","Data":"b6ed07452fa68995f016207b835d983b64b0246243e93235d089a2c175ab48ef"}
Nov 02 02:27:05.611177 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:05.610749    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bbmw9" event={"ID":"df0de6c6-be7b-4863-8f63-b6ca94e7a6e9","Type":"ContainerStarted","Data":"820a6362fc57881b4f8e0894337b04789cd118222efa1b3c679a59d5148bf9c5"}
Nov 02 02:27:05.727788 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:05.727729    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bbmw9" podStartSLOduration=1.727716179 podStartE2EDuration="1.727716179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:27:05.725625779 +0000 UTC m=+3690.629307895" watchObservedRunningTime="2026-11-02 02:27:05.727716179 +0000 UTC m=+3690.631398292"
Nov 02 02:27:06.617446 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:06.617411    5639 generic.go:356] "Generic (PLEG): container finished" podID="df0de6c6-be7b-4863-8f63-b6ca94e7a6e9" containerID="b6ed07452fa68995f016207b835d983b64b0246243e93235d089a2c175ab48ef" exitCode=0
Nov 02 02:27:06.617925 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:06.617484    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bbmw9" event={"ID":"df0de6c6-be7b-4863-8f63-b6ca94e7a6e9","Type":"ContainerDied","Data":"b6ed07452fa68995f016207b835d983b64b0246243e93235d089a2c175ab48ef"}
Nov 02 02:27:07.765713 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:07.765684    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bbmw9"
Nov 02 02:27:07.872777 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:07.872743    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df0de6c6-be7b-4863-8f63-b6ca94e7a6e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0de6c6-be7b-4863-8f63-b6ca94e7a6e9-maas-api-xks-ca\") pod \"df0de6c6-be7b-4863-8f63-b6ca94e7a6e9\" (UID: \"df0de6c6-be7b-4863-8f63-b6ca94e7a6e9\") "
Nov 02 02:27:07.872992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:07.872791    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df0de6c6-be7b-4863-8f63-b6ca94e7a6e9-kube-api-access-jchhz\" (UniqueName: \"kubernetes.io/projected/df0de6c6-be7b-4863-8f63-b6ca94e7a6e9-kube-api-access-jchhz\") pod \"df0de6c6-be7b-4863-8f63-b6ca94e7a6e9\" (UID: \"df0de6c6-be7b-4863-8f63-b6ca94e7a6e9\") "
Nov 02 02:27:07.872992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:07.872951    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df0de6c6-be7b-4863-8f63-b6ca94e7a6e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0de6c6-be7b-4863-8f63-b6ca94e7a6e9-maas-api-openshift-ca\") pod \"df0de6c6-be7b-4863-8f63-b6ca94e7a6e9\" (UID: \"df0de6c6-be7b-4863-8f63-b6ca94e7a6e9\") "
Nov 02 02:27:07.873360 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:07.873303    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0de6c6-be7b-4863-8f63-b6ca94e7a6e9-maas-api-openshift-ca" pod "df0de6c6-be7b-4863-8f63-b6ca94e7a6e9" (UID: "df0de6c6-be7b-4863-8f63-b6ca94e7a6e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:27:07.875042 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:07.875018    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df0de6c6-be7b-4863-8f63-b6ca94e7a6e9-maas-api-xks-ca" pod "df0de6c6-be7b-4863-8f63-b6ca94e7a6e9" (UID: "df0de6c6-be7b-4863-8f63-b6ca94e7a6e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:27:07.875487 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:07.875465    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df0de6c6-be7b-4863-8f63-b6ca94e7a6e9-kube-api-access-jchhz" pod "df0de6c6-be7b-4863-8f63-b6ca94e7a6e9" (UID: "df0de6c6-be7b-4863-8f63-b6ca94e7a6e9"). InnerVolumeSpecName "kube-api-access-jchhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:27:07.974002 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:07.973963    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0de6c6-be7b-4863-8f63-b6ca94e7a6e9-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:07.974002 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:07.973996    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0de6c6-be7b-4863-8f63-b6ca94e7a6e9-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:07.974002 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:07.974008    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jchhz\" (UniqueName: \"kubernetes.io/projected/df0de6c6-be7b-4863-8f63-b6ca94e7a6e9-kube-api-access-jchhz\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:08.629754 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:08.629712    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bbmw9" event={"ID":"df0de6c6-be7b-4863-8f63-b6ca94e7a6e9","Type":"ContainerDied","Data":"820a6362fc57881b4f8e0894337b04789cd118222efa1b3c679a59d5148bf9c5"}
Nov 02 02:27:08.629983 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:08.629770    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="820a6362fc57881b4f8e0894337b04789cd118222efa1b3c679a59d5148bf9c5"
Nov 02 02:27:08.629983 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:08.629725    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bbmw9"
Nov 02 02:27:17.007302 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.007209    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:27:17.375041 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.375005    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"]
Nov 02 02:27:17.375476 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.375443    5639 cpu_manager.go:414] "containerMap: removing container" podUID="df0de6c6-be7b-4863-8f63-b6ca94e7a6e9" containerName="revoke-keys"
Nov 02 02:27:17.375476 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.375457    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df0de6c6-be7b-4863-8f63-b6ca94e7a6e9" containerName="revoke-keys"
Nov 02 02:27:17.375738 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.375721    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="df0de6c6-be7b-4863-8f63-b6ca94e7a6e9" containerName="revoke-keys"
Nov 02 02:27:17.379105 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.379085    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.431523 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.431485    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-3ffe96-openshift-default-dockercfg-7vv2j\""
Nov 02 02:27:17.471049 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.471015    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-workload-certs\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.471049 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.471056    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istiod-ca-cert\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.471385 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.471213    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2z8g\" (UniqueName: \"kubernetes.io/projected/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-kube-api-access-l2z8g\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.471385 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.471253    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-workload-socket\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.471385 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.471294    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-credential-socket\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.471385 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.471371    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-token\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.471622 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.471423    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-envoy\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.471622 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.471526    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-podinfo\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.471622 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.471597    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-data\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.484873 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.484838    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"]
Nov 02 02:27:17.573326 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.573270    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-workload-certs\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.573550 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.573339    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istiod-ca-cert\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.573550 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.573397    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2z8g\" (UniqueName: \"kubernetes.io/projected/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-kube-api-access-l2z8g\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.573550 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.573438    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-workload-socket\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.573550 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.573467    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-credential-socket\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.573810 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.573601    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-token\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.573810 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.573661    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-envoy\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.573810 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.573723    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-podinfo\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.573810 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.573771    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-workload-certs\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.573810 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.573797    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-data\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.574087 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.573822    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-workload-socket\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.574087 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.573857    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-credential-socket\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.574191 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.574172    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-data\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.574694 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.574675    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istiod-ca-cert\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.576376 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.576353    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-podinfo\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.576511 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.576492    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-envoy\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.634424 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.634332    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-token\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.634568 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.634460    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2z8g\" (UniqueName: \"kubernetes.io/projected/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-kube-api-access-l2z8g\") pod \"e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") " pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:17.692421 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:17.692390    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:18.124423 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:18.124397    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"]
Nov 02 02:27:18.126454 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:27:18.126419    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e9e3aa_6453_4ada_a8f4_f59470d5c9a9.slice/crio-708e810dc762b031e41a29099cf06ba4a96f39904faace12026fd46c5209feaa WatchSource:0}: Error finding container 708e810dc762b031e41a29099cf06ba4a96f39904faace12026fd46c5209feaa: Status 404 returned error can't find the container with id 708e810dc762b031e41a29099cf06ba4a96f39904faace12026fd46c5209feaa
Nov 02 02:27:18.128668 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:18.128630    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:27:18.128781 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:18.128716    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:27:18.128781 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:18.128762    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:27:18.684238 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:18.684200    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv" event={"ID":"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9","Type":"ContainerStarted","Data":"636814bb41a1984322cd2fa6ae6228ec59077f726f39ccdbdb215ec1634d3e4d"}
Nov 02 02:27:18.684451 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:18.684256    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv" event={"ID":"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9","Type":"ContainerStarted","Data":"708e810dc762b031e41a29099cf06ba4a96f39904faace12026fd46c5209feaa"}
Nov 02 02:27:18.692633 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:18.692605    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:18.806185 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:18.806133    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv" podStartSLOduration=1.806115326 podStartE2EDuration="1.806115326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:27:18.804848058 +0000 UTC m=+3703.708530189" watchObservedRunningTime="2026-11-02 02:27:18.806115326 +0000 UTC m=+3703.709797435"
Nov 02 02:27:19.698494 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:19.698462    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:20.696296 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:20.696257    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:20.697382 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:20.697362    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:27.420607 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:27:27.420569    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:27:34.212843 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:34.212798    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd"]
Nov 02 02:27:34.221762 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:34.221731    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd"
Nov 02 02:27:34.271788 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:34.271757    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd"]
Nov 02 02:27:34.319104 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:34.319058    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-3ffe96\""
Nov 02 02:27:34.319319 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:34.319164    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-16af633a-dockercfg-q6wv7\""
Nov 02 02:27:34.345835 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:34.345800    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzsqg\" (UniqueName: \"kubernetes.io/projected/bc79cdac-eab8-4b50-9aa3-f2ce55132e0a-kube-api-access-qzsqg\") pod \"payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd\" (UID: \"bc79cdac-eab8-4b50-9aa3-f2ce55132e0a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd"
Nov 02 02:27:34.346012 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:34.345847    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc79cdac-eab8-4b50-9aa3-f2ce55132e0a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd\" (UID: \"bc79cdac-eab8-4b50-9aa3-f2ce55132e0a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd"
Nov 02 02:27:34.381573 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:34.381542    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4"]
Nov 02 02:27:34.385430 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:34.385407    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4"
Nov 02 02:27:34.445660 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:34.445626    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4"]
Nov 02 02:27:34.447050 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:34.447022    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzsqg\" (UniqueName: \"kubernetes.io/projected/bc79cdac-eab8-4b50-9aa3-f2ce55132e0a-kube-api-access-qzsqg\") pod \"payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd\" (UID: \"bc79cdac-eab8-4b50-9aa3-f2ce55132e0a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd"
Nov 02 02:27:34.447189 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:34.447060    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc79cdac-eab8-4b50-9aa3-f2ce55132e0a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd\" (UID: \"bc79cdac-eab8-4b50-9aa3-f2ce55132e0a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd"
Nov 02 02:27:34.447189 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:34.447108    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2184c94a-607a-45ea-bf1f-fcba87b1259b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4\" (UID: \"2184c94a-607a-45ea-bf1f-fcba87b1259b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4"
Nov 02 02:27:34.447383 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:34.447324    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhmv8\" (UniqueName: \"kubernetes.io/projected/2184c94a-607a-45ea-bf1f-fcba87b1259b-kube-api-access-lhmv8\") pod \"payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4\" (UID: \"2184c94a-607a-45ea-bf1f-fcba87b1259b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4"
Nov 02 02:27:34.447855 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:34.447833    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc79cdac-eab8-4b50-9aa3-f2ce55132e0a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd\" (UID: \"bc79cdac-eab8-4b50-9aa3-f2ce55132e0a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd"
Nov 02 02:27:34.516392 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:34.510451    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzsqg\" (UniqueName: \"kubernetes.io/projected/bc79cdac-eab8-4b50-9aa3-f2ce55132e0a-kube-api-access-qzsqg\") pod \"payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd\" (UID: \"bc79cdac-eab8-4b50-9aa3-f2ce55132e0a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd"
Nov 02 02:27:34.535986 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:34.535952    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd"
Nov 02 02:27:34.548476 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:34.548447    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2184c94a-607a-45ea-bf1f-fcba87b1259b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4\" (UID: \"2184c94a-607a-45ea-bf1f-fcba87b1259b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4"
Nov 02 02:27:34.548594 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:34.548557    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhmv8\" (UniqueName: \"kubernetes.io/projected/2184c94a-607a-45ea-bf1f-fcba87b1259b-kube-api-access-lhmv8\") pod \"payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4\" (UID: \"2184c94a-607a-45ea-bf1f-fcba87b1259b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4"
Nov 02 02:27:34.549155 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:34.549130    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2184c94a-607a-45ea-bf1f-fcba87b1259b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4\" (UID: \"2184c94a-607a-45ea-bf1f-fcba87b1259b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4"
Nov 02 02:27:34.611260 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:34.611188    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhmv8\" (UniqueName: \"kubernetes.io/projected/2184c94a-607a-45ea-bf1f-fcba87b1259b-kube-api-access-lhmv8\") pod \"payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4\" (UID: \"2184c94a-607a-45ea-bf1f-fcba87b1259b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4"
Nov 02 02:27:34.698569 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:34.698532    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4"
Nov 02 02:27:34.910385 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:34.910361    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd"]
Nov 02 02:27:34.912517 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:27:34.912493    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc79cdac_eab8_4b50_9aa3_f2ce55132e0a.slice/crio-eff78aecdf48cf49cc3ee7a5924d9569122a6214fde1ddd325bfedf73919cce0 WatchSource:0}: Error finding container eff78aecdf48cf49cc3ee7a5924d9569122a6214fde1ddd325bfedf73919cce0: Status 404 returned error can't find the container with id eff78aecdf48cf49cc3ee7a5924d9569122a6214fde1ddd325bfedf73919cce0
Nov 02 02:27:35.072351 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:35.072085    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4"]
Nov 02 02:27:35.074657 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:27:35.074628    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2184c94a_607a_45ea_bf1f_fcba87b1259b.slice/crio-ef8ec294bb9c8f5ff86fa9c902a0ec6e3c69a4839e4700b8b68ab8f5555bd494 WatchSource:0}: Error finding container ef8ec294bb9c8f5ff86fa9c902a0ec6e3c69a4839e4700b8b68ab8f5555bd494: Status 404 returned error can't find the container with id ef8ec294bb9c8f5ff86fa9c902a0ec6e3c69a4839e4700b8b68ab8f5555bd494
Nov 02 02:27:35.804905 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:35.804865    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4" event={"ID":"2184c94a-607a-45ea-bf1f-fcba87b1259b","Type":"ContainerStarted","Data":"732d640743dd35da5931a7695a2d8121002e9ed33f99aeaae17ef018e689915f"}
Nov 02 02:27:35.805600 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:35.804933    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4" event={"ID":"2184c94a-607a-45ea-bf1f-fcba87b1259b","Type":"ContainerStarted","Data":"ef8ec294bb9c8f5ff86fa9c902a0ec6e3c69a4839e4700b8b68ab8f5555bd494"}
Nov 02 02:27:35.805829 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:35.805805    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4"
Nov 02 02:27:35.808991 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:35.808959    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd" event={"ID":"bc79cdac-eab8-4b50-9aa3-f2ce55132e0a","Type":"ContainerStarted","Data":"a7a48e157a60cac22f7ca18fc5dfde8ba6bc7081a40550723a671e2f6d6647ba"}
Nov 02 02:27:35.810194 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:35.809015    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd" event={"ID":"bc79cdac-eab8-4b50-9aa3-f2ce55132e0a","Type":"ContainerStarted","Data":"eff78aecdf48cf49cc3ee7a5924d9569122a6214fde1ddd325bfedf73919cce0"}
Nov 02 02:27:35.810194 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:35.809707    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd"
Nov 02 02:27:35.921952 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:35.921894    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4" podStartSLOduration=1.9218796280000001 podStartE2EDuration="1.921879628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:27:35.920915024 +0000 UTC m=+3720.824597138" watchObservedRunningTime="2026-11-02 02:27:35.921879628 +0000 UTC m=+3720.825561740"
Nov 02 02:27:36.017620 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:36.017581    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:27:36.039378 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:36.039327    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd" podStartSLOduration=2.039311639 podStartE2EDuration="2.039311639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:27:36.037885251 +0000 UTC m=+3720.941567365" watchObservedRunningTime="2026-11-02 02:27:36.039311639 +0000 UTC m=+3720.942993805"
Nov 02 02:27:39.686944 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:39.686908    5639 scope.go:122] "RemoveContainer" containerID="1be628a4c5dc3f4450a980c4104f39253708dd4545e5c2ca36f13f8b280b36c4"
Nov 02 02:27:39.696725 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:39.696701    5639 scope.go:122] "RemoveContainer" containerID="d4b100017bcbc4c6479e3a26705192c207b9ded0cd40d30fbec8c7c41bf18241"
Nov 02 02:27:45.872744 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:45.872647    5639 generic.go:356] "Generic (PLEG): container finished" podID="89e9e3aa-6453-4ada-a8f4-f59470d5c9a9" containerID="636814bb41a1984322cd2fa6ae6228ec59077f726f39ccdbdb215ec1634d3e4d" exitCode=0
Nov 02 02:27:45.872744 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:45.872722    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv" event={"ID":"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9","Type":"ContainerDied","Data":"636814bb41a1984322cd2fa6ae6228ec59077f726f39ccdbdb215ec1634d3e4d"}
Nov 02 02:27:45.873236 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:45.873151    5639 scope.go:122] "RemoveContainer" containerID="636814bb41a1984322cd2fa6ae6228ec59077f726f39ccdbdb215ec1634d3e4d"
Nov 02 02:27:45.874422 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:45.874391    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:27:45.874521 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:45.874465    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:27:45.874521 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:45.874501    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:27:46.889973 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:46.889937    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv" event={"ID":"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9","Type":"ContainerStarted","Data":"2f905d524dd514268a3d8e23df9d75224608d671270c76dfc8196ff43506691f"}
Nov 02 02:27:47.693650 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:47.693606    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:47.694705 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:47.694680    5639 patch_prober.go:29] interesting pod/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body=
Nov 02 02:27:47.694839 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:47.694726    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv" podUID="89e9e3aa-6453-4ada-a8f4-f59470d5c9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused"
Nov 02 02:27:47.823040 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:47.823010    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd"
Nov 02 02:27:47.823241 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:47.823173    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4"
Nov 02 02:27:48.692834 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:48.692796    5639 patch_prober.go:29] interesting pod/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body=
Nov 02 02:27:48.693305 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:48.692856    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv" podUID="89e9e3aa-6453-4ada-a8f4-f59470d5c9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused"
Nov 02 02:27:49.692862 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:49.692819    5639 patch_prober.go:29] interesting pod/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body=
Nov 02 02:27:49.693320 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:49.692882    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv" podUID="89e9e3aa-6453-4ada-a8f4-f59470d5c9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused"
Nov 02 02:27:50.692845 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:50.692805    5639 patch_prober.go:29] interesting pod/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body=
Nov 02 02:27:50.693052 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:50.692864    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv" podUID="89e9e3aa-6453-4ada-a8f4-f59470d5c9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused"
Nov 02 02:27:50.696965 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:50.696940    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:50.992467 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:50.992382    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"]
Nov 02 02:27:50.995870 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:50.995848    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"
Nov 02 02:27:51.031926 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.031878    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-tls-certs\") pod \"simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv\" (UID: \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\") " pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"
Nov 02 02:27:51.032121 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.032021    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-dshm\") pod \"simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv\" (UID: \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\") " pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"
Nov 02 02:27:51.032121 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.032065    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v2mx\" (UniqueName: \"kubernetes.io/projected/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-kube-api-access-7v2mx\") pod \"simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv\" (UID: \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\") " pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"
Nov 02 02:27:51.032308 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.032254    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-home\") pod \"simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv\" (UID: \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\") " pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"
Nov 02 02:27:51.032383 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.032312    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-tmp-dir\") pod \"simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv\" (UID: \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\") " pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"
Nov 02 02:27:51.032383 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.032333    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-model-cache\") pod \"simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv\" (UID: \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\") " pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"
Nov 02 02:27:51.097714 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.097670    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-3ffe96\"/\"default-dockercfg-m8z8b\""
Nov 02 02:27:51.098184 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.097788    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-3ffe96\"/\"kube-root-ca.crt\""
Nov 02 02:27:51.099173 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.098057    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-3ffe96\"/\"simulator-single-main-3ffe96-kserve-self-signed-certs\""
Nov 02 02:27:51.099369 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.098138    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-3ffe96\"/\"openshift-service-ca.crt\""
Nov 02 02:27:51.101733 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.101704    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"]
Nov 02 02:27:51.133096 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.133068    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-home\") pod \"simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv\" (UID: \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\") " pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"
Nov 02 02:27:51.133312 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.133103    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-tmp-dir\") pod \"simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv\" (UID: \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\") " pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"
Nov 02 02:27:51.133312 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.133129    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-model-cache\") pod \"simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv\" (UID: \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\") " pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"
Nov 02 02:27:51.133312 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.133152    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-tls-certs\") pod \"simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv\" (UID: \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\") " pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"
Nov 02 02:27:51.133312 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.133208    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-dshm\") pod \"simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv\" (UID: \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\") " pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"
Nov 02 02:27:51.133312 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.133233    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v2mx\" (UniqueName: \"kubernetes.io/projected/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-kube-api-access-7v2mx\") pod \"simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv\" (UID: \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\") " pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"
Nov 02 02:27:51.133647 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.133624    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-tmp-dir\") pod \"simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv\" (UID: \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\") " pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"
Nov 02 02:27:51.133714 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.133633    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-model-cache\") pod \"simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv\" (UID: \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\") " pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"
Nov 02 02:27:51.133794 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.133774    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-home\") pod \"simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv\" (UID: \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\") " pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"
Nov 02 02:27:51.136171 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.135790    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-dshm\") pod \"simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv\" (UID: \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\") " pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"
Nov 02 02:27:51.136171 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.136042    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-tls-certs\") pod \"simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv\" (UID: \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\") " pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"
Nov 02 02:27:51.194899 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.194868    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v2mx\" (UniqueName: \"kubernetes.io/projected/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-kube-api-access-7v2mx\") pod \"simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv\" (UID: \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\") " pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"
Nov 02 02:27:51.308368 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.308246    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"
Nov 02 02:27:51.693392 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.693353    5639 patch_prober.go:29] interesting pod/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body=
Nov 02 02:27:51.693817 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.693411    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv" podUID="89e9e3aa-6453-4ada-a8f4-f59470d5c9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused"
Nov 02 02:27:51.700779 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.700756    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"]
Nov 02 02:27:51.703104 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:27:51.703076    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a0a1833_f7b7_4cb2_932a_3006c4c66c1b.slice/crio-c6c16dee02f25ab6086297635edc6ba9d8a2b82da6070c846d0b2b6eff4c20d4 WatchSource:0}: Error finding container c6c16dee02f25ab6086297635edc6ba9d8a2b82da6070c846d0b2b6eff4c20d4: Status 404 returned error can't find the container with id c6c16dee02f25ab6086297635edc6ba9d8a2b82da6070c846d0b2b6eff4c20d4
Nov 02 02:27:51.922031 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.921984    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv" event={"ID":"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b","Type":"ContainerStarted","Data":"4cf35d78433ce7ce38035e3e18789ff1c1c39f929f2764fceefcd684fdcfddb4"}
Nov 02 02:27:51.922237 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:51.922047    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv" event={"ID":"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b","Type":"ContainerStarted","Data":"c6c16dee02f25ab6086297635edc6ba9d8a2b82da6070c846d0b2b6eff4c20d4"}
Nov 02 02:27:52.039421 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:52.039312    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv" podStartSLOduration=2.039297585 podStartE2EDuration="2.039297585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:27:52.038469058 +0000 UTC m=+3736.942151189" watchObservedRunningTime="2026-11-02 02:27:52.039297585 +0000 UTC m=+3736.942979690"
Nov 02 02:27:52.309187 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:52.309090    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"
Nov 02 02:27:52.697450 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:52.697422    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:52.698466 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:52.698444    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:27:55.700614 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:27:55.700580    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:27:57.421171 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:27:57.421136    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:28:01.308472 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:01.308430    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"
Nov 02 02:28:01.321040 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:01.321010    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"
Nov 02 02:28:01.991555 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:01.991525    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"
Nov 02 02:28:10.118968 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:10.118929    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:28:25.210914 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:25.210878    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d63dc943-ft5fl"]
Nov 02 02:28:25.215072 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:25.215049    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63dc943-ft5fl"
Nov 02 02:28:25.269976 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:25.269808    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:28:25.290577 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:25.290547    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b3706cf-d187-4c88-bafd-c9c85738aeb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d63dc943-ft5fl\" (UID: \"8b3706cf-d187-4c88-bafd-c9c85738aeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63dc943-ft5fl"
Nov 02 02:28:25.290733 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:25.290585    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b3706cf-d187-4c88-bafd-c9c85738aeb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d63dc943-ft5fl\" (UID: \"8b3706cf-d187-4c88-bafd-c9c85738aeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63dc943-ft5fl"
Nov 02 02:28:25.290733 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:25.290705    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lppb\" (UniqueName: \"kubernetes.io/projected/8b3706cf-d187-4c88-bafd-c9c85738aeb5-kube-api-access-7lppb\") pod \"maas-api-revoke-sub-d63dc943-ft5fl\" (UID: \"8b3706cf-d187-4c88-bafd-c9c85738aeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63dc943-ft5fl"
Nov 02 02:28:25.319496 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:25.319460    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:28:25.319496 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:25.319488    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:28:25.319708 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:25.319505    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:28:25.321546 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:25.321522    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d63dc943-ft5fl"]
Nov 02 02:28:25.392199 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:25.392166    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b3706cf-d187-4c88-bafd-c9c85738aeb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d63dc943-ft5fl\" (UID: \"8b3706cf-d187-4c88-bafd-c9c85738aeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63dc943-ft5fl"
Nov 02 02:28:25.392401 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:25.392205    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b3706cf-d187-4c88-bafd-c9c85738aeb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d63dc943-ft5fl\" (UID: \"8b3706cf-d187-4c88-bafd-c9c85738aeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63dc943-ft5fl"
Nov 02 02:28:25.392401 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:25.392327    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lppb\" (UniqueName: \"kubernetes.io/projected/8b3706cf-d187-4c88-bafd-c9c85738aeb5-kube-api-access-7lppb\") pod \"maas-api-revoke-sub-d63dc943-ft5fl\" (UID: \"8b3706cf-d187-4c88-bafd-c9c85738aeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63dc943-ft5fl"
Nov 02 02:28:25.392936 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:25.392912    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b3706cf-d187-4c88-bafd-c9c85738aeb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d63dc943-ft5fl\" (UID: \"8b3706cf-d187-4c88-bafd-c9c85738aeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63dc943-ft5fl"
Nov 02 02:28:25.394762 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:25.394739    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b3706cf-d187-4c88-bafd-c9c85738aeb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d63dc943-ft5fl\" (UID: \"8b3706cf-d187-4c88-bafd-c9c85738aeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63dc943-ft5fl"
Nov 02 02:28:25.453423 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:25.453393    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lppb\" (UniqueName: \"kubernetes.io/projected/8b3706cf-d187-4c88-bafd-c9c85738aeb5-kube-api-access-7lppb\") pod \"maas-api-revoke-sub-d63dc943-ft5fl\" (UID: \"8b3706cf-d187-4c88-bafd-c9c85738aeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63dc943-ft5fl"
Nov 02 02:28:25.528985 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:25.528913    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63dc943-ft5fl"
Nov 02 02:28:25.901092 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:25.900941    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d63dc943-ft5fl"]
Nov 02 02:28:25.904171 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:28:25.904142    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b3706cf_d187_4c88_bafd_c9c85738aeb5.slice/crio-3b9a18e6594d090e581154deb568754b485e7d737f37f3dd8db74e7e7f217cec WatchSource:0}: Error finding container 3b9a18e6594d090e581154deb568754b485e7d737f37f3dd8db74e7e7f217cec: Status 404 returned error can't find the container with id 3b9a18e6594d090e581154deb568754b485e7d737f37f3dd8db74e7e7f217cec
Nov 02 02:28:26.110305 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:26.110262    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63dc943-ft5fl" event={"ID":"8b3706cf-d187-4c88-bafd-c9c85738aeb5","Type":"ContainerStarted","Data":"3b9a18e6594d090e581154deb568754b485e7d737f37f3dd8db74e7e7f217cec"}
Nov 02 02:28:27.118327 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:27.118274    5639 generic.go:356] "Generic (PLEG): container finished" podID="8b3706cf-d187-4c88-bafd-c9c85738aeb5" containerID="1aeb02100fc2a57ea51da832262534b705a4c352e0371a24fdfd165e420ea04e" exitCode=0
Nov 02 02:28:27.118759 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:27.118387    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63dc943-ft5fl" event={"ID":"8b3706cf-d187-4c88-bafd-c9c85738aeb5","Type":"ContainerDied","Data":"1aeb02100fc2a57ea51da832262534b705a4c352e0371a24fdfd165e420ea04e"}
Nov 02 02:28:27.421094 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:28:27.421005    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:28:28.276578 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:28.276552    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63dc943-ft5fl"
Nov 02 02:28:28.327105 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:28.327070    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b3706cf-d187-4c88-bafd-c9c85738aeb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b3706cf-d187-4c88-bafd-c9c85738aeb5-maas-api-xks-ca\") pod \"8b3706cf-d187-4c88-bafd-c9c85738aeb5\" (UID: \"8b3706cf-d187-4c88-bafd-c9c85738aeb5\") "
Nov 02 02:28:28.327294 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:28.327149    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b3706cf-d187-4c88-bafd-c9c85738aeb5-kube-api-access-7lppb\" (UniqueName: \"kubernetes.io/projected/8b3706cf-d187-4c88-bafd-c9c85738aeb5-kube-api-access-7lppb\") pod \"8b3706cf-d187-4c88-bafd-c9c85738aeb5\" (UID: \"8b3706cf-d187-4c88-bafd-c9c85738aeb5\") "
Nov 02 02:28:28.327294 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:28.327208    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b3706cf-d187-4c88-bafd-c9c85738aeb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b3706cf-d187-4c88-bafd-c9c85738aeb5-maas-api-openshift-ca\") pod \"8b3706cf-d187-4c88-bafd-c9c85738aeb5\" (UID: \"8b3706cf-d187-4c88-bafd-c9c85738aeb5\") "
Nov 02 02:28:28.327557 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:28.327532    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b3706cf-d187-4c88-bafd-c9c85738aeb5-maas-api-openshift-ca" pod "8b3706cf-d187-4c88-bafd-c9c85738aeb5" (UID: "8b3706cf-d187-4c88-bafd-c9c85738aeb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:28:28.327639 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:28.327617    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b3706cf-d187-4c88-bafd-c9c85738aeb5-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:28:28.329520 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:28.329489    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b3706cf-d187-4c88-bafd-c9c85738aeb5-maas-api-xks-ca" pod "8b3706cf-d187-4c88-bafd-c9c85738aeb5" (UID: "8b3706cf-d187-4c88-bafd-c9c85738aeb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:28:28.329644 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:28.329569    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b3706cf-d187-4c88-bafd-c9c85738aeb5-kube-api-access-7lppb" pod "8b3706cf-d187-4c88-bafd-c9c85738aeb5" (UID: "8b3706cf-d187-4c88-bafd-c9c85738aeb5"). InnerVolumeSpecName "kube-api-access-7lppb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:28:28.428727 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:28.428643    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b3706cf-d187-4c88-bafd-c9c85738aeb5-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:28:28.428727 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:28.428679    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lppb\" (UniqueName: \"kubernetes.io/projected/8b3706cf-d187-4c88-bafd-c9c85738aeb5-kube-api-access-7lppb\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:28:29.130424 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:29.130398    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63dc943-ft5fl"
Nov 02 02:28:29.130687 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:29.130391    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63dc943-ft5fl" event={"ID":"8b3706cf-d187-4c88-bafd-c9c85738aeb5","Type":"ContainerDied","Data":"3b9a18e6594d090e581154deb568754b485e7d737f37f3dd8db74e7e7f217cec"}
Nov 02 02:28:29.130846 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:29.130739    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b9a18e6594d090e581154deb568754b485e7d737f37f3dd8db74e7e7f217cec"
Nov 02 02:28:36.882180 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:36.882146    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5"]
Nov 02 02:28:36.882869 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:36.882639    5639 cpu_manager.go:414] "containerMap: removing container" podUID="8b3706cf-d187-4c88-bafd-c9c85738aeb5" containerName="revoke-keys"
Nov 02 02:28:36.882869 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:36.882652    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b3706cf-d187-4c88-bafd-c9c85738aeb5" containerName="revoke-keys"
Nov 02 02:28:36.882869 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:36.882726    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b3706cf-d187-4c88-bafd-c9c85738aeb5" containerName="revoke-keys"
Nov 02 02:28:36.886016 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:36.885998    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5"
Nov 02 02:28:36.939382 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:36.939354    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:28:36.939382 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:36.939356    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:28:36.939382 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:36.939361    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:28:36.988957 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:36.988928    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:28:36.991154 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:36.991131    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5"]
Nov 02 02:28:37.019564 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:37.019533    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d229e732-b43a-45e8-8ea4-93302a4274b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5\" (UID: \"d229e732-b43a-45e8-8ea4-93302a4274b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5"
Nov 02 02:28:37.019747 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:37.019604    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d229e732-b43a-45e8-8ea4-93302a4274b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5\" (UID: \"d229e732-b43a-45e8-8ea4-93302a4274b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5"
Nov 02 02:28:37.019747 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:37.019671    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfvcr\" (UniqueName: \"kubernetes.io/projected/d229e732-b43a-45e8-8ea4-93302a4274b1-kube-api-access-mfvcr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5\" (UID: \"d229e732-b43a-45e8-8ea4-93302a4274b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5"
Nov 02 02:28:37.120965 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:37.120926    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d229e732-b43a-45e8-8ea4-93302a4274b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5\" (UID: \"d229e732-b43a-45e8-8ea4-93302a4274b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5"
Nov 02 02:28:37.120965 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:37.120955    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfvcr\" (UniqueName: \"kubernetes.io/projected/d229e732-b43a-45e8-8ea4-93302a4274b1-kube-api-access-mfvcr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5\" (UID: \"d229e732-b43a-45e8-8ea4-93302a4274b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5"
Nov 02 02:28:37.121174 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:37.121034    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d229e732-b43a-45e8-8ea4-93302a4274b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5\" (UID: \"d229e732-b43a-45e8-8ea4-93302a4274b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5"
Nov 02 02:28:37.121702 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:37.121670    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d229e732-b43a-45e8-8ea4-93302a4274b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5\" (UID: \"d229e732-b43a-45e8-8ea4-93302a4274b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5"
Nov 02 02:28:37.123566 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:37.123536    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d229e732-b43a-45e8-8ea4-93302a4274b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5\" (UID: \"d229e732-b43a-45e8-8ea4-93302a4274b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5"
Nov 02 02:28:37.182845 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:37.182779    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfvcr\" (UniqueName: \"kubernetes.io/projected/d229e732-b43a-45e8-8ea4-93302a4274b1-kube-api-access-mfvcr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5\" (UID: \"d229e732-b43a-45e8-8ea4-93302a4274b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5"
Nov 02 02:28:37.198717 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:37.198688    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5"
Nov 02 02:28:37.575167 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:37.575145    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5"]
Nov 02 02:28:37.577214 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:28:37.577182    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd229e732_b43a_45e8_8ea4_93302a4274b1.slice/crio-b763663a85a1b67b03fa818d3ddde19b12e2c8660af91a8bdf9e5679e7fb799d WatchSource:0}: Error finding container b763663a85a1b67b03fa818d3ddde19b12e2c8660af91a8bdf9e5679e7fb799d: Status 404 returned error can't find the container with id b763663a85a1b67b03fa818d3ddde19b12e2c8660af91a8bdf9e5679e7fb799d
Nov 02 02:28:38.179034 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:38.178944    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5" event={"ID":"d229e732-b43a-45e8-8ea4-93302a4274b1","Type":"ContainerStarted","Data":"c1f2d3712f7c44e4b00f11a0041c14a6e28bdd4f7201ae2f932b1dc502e3f8ab"}
Nov 02 02:28:38.179034 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:38.178995    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5" event={"ID":"d229e732-b43a-45e8-8ea4-93302a4274b1","Type":"ContainerStarted","Data":"b763663a85a1b67b03fa818d3ddde19b12e2c8660af91a8bdf9e5679e7fb799d"}
Nov 02 02:28:38.214163 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:38.214129    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:28:38.296199 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:38.296147    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5" podStartSLOduration=2.296131627 podStartE2EDuration="2.296131627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:28:38.294915614 +0000 UTC m=+3783.198597729" watchObservedRunningTime="2026-11-02 02:28:38.296131627 +0000 UTC m=+3783.199813740"
Nov 02 02:28:39.185774 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:39.185736    5639 generic.go:356] "Generic (PLEG): container finished" podID="d229e732-b43a-45e8-8ea4-93302a4274b1" containerID="c1f2d3712f7c44e4b00f11a0041c14a6e28bdd4f7201ae2f932b1dc502e3f8ab" exitCode=0
Nov 02 02:28:39.186196 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:39.185778    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5" event={"ID":"d229e732-b43a-45e8-8ea4-93302a4274b1","Type":"ContainerDied","Data":"c1f2d3712f7c44e4b00f11a0041c14a6e28bdd4f7201ae2f932b1dc502e3f8ab"}
Nov 02 02:28:39.722179 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:39.722150    5639 scope.go:122] "RemoveContainer" containerID="d558876c79c4424b0709fdbf1ca358c2e3621effd2af89145a5b118446b3ee1b"
Nov 02 02:28:39.731415 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:39.731393    5639 scope.go:122] "RemoveContainer" containerID="4dfb1cbaa982e6808f3d6ed3b53505c057df14b361d49c3957d6f852e9a781bf"
Nov 02 02:28:40.321419 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:40.321395    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5"
Nov 02 02:28:40.455192 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:40.455117    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d229e732-b43a-45e8-8ea4-93302a4274b1-kube-api-access-mfvcr\" (UniqueName: \"kubernetes.io/projected/d229e732-b43a-45e8-8ea4-93302a4274b1-kube-api-access-mfvcr\") pod \"d229e732-b43a-45e8-8ea4-93302a4274b1\" (UID: \"d229e732-b43a-45e8-8ea4-93302a4274b1\") "
Nov 02 02:28:40.455387 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:40.455223    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d229e732-b43a-45e8-8ea4-93302a4274b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d229e732-b43a-45e8-8ea4-93302a4274b1-maas-api-xks-ca\") pod \"d229e732-b43a-45e8-8ea4-93302a4274b1\" (UID: \"d229e732-b43a-45e8-8ea4-93302a4274b1\") "
Nov 02 02:28:40.455387 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:40.455353    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d229e732-b43a-45e8-8ea4-93302a4274b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d229e732-b43a-45e8-8ea4-93302a4274b1-maas-api-openshift-ca\") pod \"d229e732-b43a-45e8-8ea4-93302a4274b1\" (UID: \"d229e732-b43a-45e8-8ea4-93302a4274b1\") "
Nov 02 02:28:40.455742 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:40.455718    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d229e732-b43a-45e8-8ea4-93302a4274b1-maas-api-openshift-ca" pod "d229e732-b43a-45e8-8ea4-93302a4274b1" (UID: "d229e732-b43a-45e8-8ea4-93302a4274b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:28:40.457417 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:40.457386    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d229e732-b43a-45e8-8ea4-93302a4274b1-kube-api-access-mfvcr" pod "d229e732-b43a-45e8-8ea4-93302a4274b1" (UID: "d229e732-b43a-45e8-8ea4-93302a4274b1"). InnerVolumeSpecName "kube-api-access-mfvcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:28:40.457523 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:40.457446    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d229e732-b43a-45e8-8ea4-93302a4274b1-maas-api-xks-ca" pod "d229e732-b43a-45e8-8ea4-93302a4274b1" (UID: "d229e732-b43a-45e8-8ea4-93302a4274b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:28:40.556602 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:40.556552    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d229e732-b43a-45e8-8ea4-93302a4274b1-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:28:40.556602 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:40.556595    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfvcr\" (UniqueName: \"kubernetes.io/projected/d229e732-b43a-45e8-8ea4-93302a4274b1-kube-api-access-mfvcr\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:28:40.556602 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:40.556606    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d229e732-b43a-45e8-8ea4-93302a4274b1-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:28:41.197553 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:41.197522    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5"
Nov 02 02:28:41.197740 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:41.197567    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5" event={"ID":"d229e732-b43a-45e8-8ea4-93302a4274b1","Type":"ContainerDied","Data":"b763663a85a1b67b03fa818d3ddde19b12e2c8660af91a8bdf9e5679e7fb799d"}
Nov 02 02:28:41.197740 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:41.197611    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b763663a85a1b67b03fa818d3ddde19b12e2c8660af91a8bdf9e5679e7fb799d"
Nov 02 02:28:41.350142 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:41.350105    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"]
Nov 02 02:28:48.361909 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:48.361814    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd"]
Nov 02 02:28:48.467093 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:48.467048    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4"]
Nov 02 02:28:53.363186 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:53.363122    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd" podUID="bc79cdac-eab8-4b50-9aa3-f2ce55132e0a" containerName="payload-processing" containerID="cri-o://a7a48e157a60cac22f7ca18fc5dfde8ba6bc7081a40550723a671e2f6d6647ba" gracePeriod=25
Nov 02 02:28:53.468171 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:53.468109    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4" podUID="2184c94a-607a-45ea-bf1f-fcba87b1259b" containerName="payload-pre-processing" containerID="cri-o://732d640743dd35da5931a7695a2d8121002e9ed33f99aeaae17ef018e689915f" gracePeriod=25
Nov 02 02:28:53.615900 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:53.615834    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd"
Nov 02 02:28:53.731476 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:53.731451    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4"
Nov 02 02:28:53.795877 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:53.795846    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc79cdac-eab8-4b50-9aa3-f2ce55132e0a-kube-api-access-qzsqg\" (UniqueName: \"kubernetes.io/projected/bc79cdac-eab8-4b50-9aa3-f2ce55132e0a-kube-api-access-qzsqg\") pod \"bc79cdac-eab8-4b50-9aa3-f2ce55132e0a\" (UID: \"bc79cdac-eab8-4b50-9aa3-f2ce55132e0a\") "
Nov 02 02:28:53.796063 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:53.795967    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc79cdac-eab8-4b50-9aa3-f2ce55132e0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc79cdac-eab8-4b50-9aa3-f2ce55132e0a-plugins-config-volume\") pod \"bc79cdac-eab8-4b50-9aa3-f2ce55132e0a\" (UID: \"bc79cdac-eab8-4b50-9aa3-f2ce55132e0a\") "
Nov 02 02:28:53.796380 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:53.796358    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc79cdac-eab8-4b50-9aa3-f2ce55132e0a-plugins-config-volume" pod "bc79cdac-eab8-4b50-9aa3-f2ce55132e0a" (UID: "bc79cdac-eab8-4b50-9aa3-f2ce55132e0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:28:53.798153 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:53.798135    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc79cdac-eab8-4b50-9aa3-f2ce55132e0a-kube-api-access-qzsqg" pod "bc79cdac-eab8-4b50-9aa3-f2ce55132e0a" (UID: "bc79cdac-eab8-4b50-9aa3-f2ce55132e0a"). InnerVolumeSpecName "kube-api-access-qzsqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:28:53.897014 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:53.896938    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2184c94a-607a-45ea-bf1f-fcba87b1259b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2184c94a-607a-45ea-bf1f-fcba87b1259b-plugins-config-volume\") pod \"2184c94a-607a-45ea-bf1f-fcba87b1259b\" (UID: \"2184c94a-607a-45ea-bf1f-fcba87b1259b\") "
Nov 02 02:28:53.897014 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:53.897008    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2184c94a-607a-45ea-bf1f-fcba87b1259b-kube-api-access-lhmv8\" (UniqueName: \"kubernetes.io/projected/2184c94a-607a-45ea-bf1f-fcba87b1259b-kube-api-access-lhmv8\") pod \"2184c94a-607a-45ea-bf1f-fcba87b1259b\" (UID: \"2184c94a-607a-45ea-bf1f-fcba87b1259b\") "
Nov 02 02:28:53.897244 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:53.897233    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzsqg\" (UniqueName: \"kubernetes.io/projected/bc79cdac-eab8-4b50-9aa3-f2ce55132e0a-kube-api-access-qzsqg\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:28:53.897303 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:53.897246    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc79cdac-eab8-4b50-9aa3-f2ce55132e0a-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:28:53.897343 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:53.897275    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2184c94a-607a-45ea-bf1f-fcba87b1259b-plugins-config-volume" pod "2184c94a-607a-45ea-bf1f-fcba87b1259b" (UID: "2184c94a-607a-45ea-bf1f-fcba87b1259b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:28:53.899178 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:53.899159    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2184c94a-607a-45ea-bf1f-fcba87b1259b-kube-api-access-lhmv8" pod "2184c94a-607a-45ea-bf1f-fcba87b1259b" (UID: "2184c94a-607a-45ea-bf1f-fcba87b1259b"). InnerVolumeSpecName "kube-api-access-lhmv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:28:53.998447 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:53.998417    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2184c94a-607a-45ea-bf1f-fcba87b1259b-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:28:53.998447 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:53.998444    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhmv8\" (UniqueName: \"kubernetes.io/projected/2184c94a-607a-45ea-bf1f-fcba87b1259b-kube-api-access-lhmv8\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:28:54.266711 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:54.266620    5639 generic.go:356] "Generic (PLEG): container finished" podID="2184c94a-607a-45ea-bf1f-fcba87b1259b" containerID="732d640743dd35da5931a7695a2d8121002e9ed33f99aeaae17ef018e689915f" exitCode=0
Nov 02 02:28:54.266711 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:54.266673    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4"
Nov 02 02:28:54.266950 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:54.266704    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4" event={"ID":"2184c94a-607a-45ea-bf1f-fcba87b1259b","Type":"ContainerDied","Data":"732d640743dd35da5931a7695a2d8121002e9ed33f99aeaae17ef018e689915f"}
Nov 02 02:28:54.266950 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:54.266758    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4" event={"ID":"2184c94a-607a-45ea-bf1f-fcba87b1259b","Type":"ContainerDied","Data":"ef8ec294bb9c8f5ff86fa9c902a0ec6e3c69a4839e4700b8b68ab8f5555bd494"}
Nov 02 02:28:54.266950 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:54.266783    5639 scope.go:122] "RemoveContainer" containerID="732d640743dd35da5931a7695a2d8121002e9ed33f99aeaae17ef018e689915f"
Nov 02 02:28:54.268101 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:54.268078    5639 generic.go:356] "Generic (PLEG): container finished" podID="bc79cdac-eab8-4b50-9aa3-f2ce55132e0a" containerID="a7a48e157a60cac22f7ca18fc5dfde8ba6bc7081a40550723a671e2f6d6647ba" exitCode=0
Nov 02 02:28:54.268218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:54.268108    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd" event={"ID":"bc79cdac-eab8-4b50-9aa3-f2ce55132e0a","Type":"ContainerDied","Data":"a7a48e157a60cac22f7ca18fc5dfde8ba6bc7081a40550723a671e2f6d6647ba"}
Nov 02 02:28:54.268218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:54.268124    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd"
Nov 02 02:28:54.268218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:54.268158    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd" event={"ID":"bc79cdac-eab8-4b50-9aa3-f2ce55132e0a","Type":"ContainerDied","Data":"eff78aecdf48cf49cc3ee7a5924d9569122a6214fde1ddd325bfedf73919cce0"}
Nov 02 02:28:54.276525 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:54.276505    5639 scope.go:122] "RemoveContainer" containerID="732d640743dd35da5931a7695a2d8121002e9ed33f99aeaae17ef018e689915f"
Nov 02 02:28:54.276845 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:28:54.276824    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"732d640743dd35da5931a7695a2d8121002e9ed33f99aeaae17ef018e689915f\": container with ID starting with 732d640743dd35da5931a7695a2d8121002e9ed33f99aeaae17ef018e689915f not found: ID does not exist" containerID="732d640743dd35da5931a7695a2d8121002e9ed33f99aeaae17ef018e689915f"
Nov 02 02:28:54.276952 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:54.276851    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"732d640743dd35da5931a7695a2d8121002e9ed33f99aeaae17ef018e689915f"} err="failed to get container status \"732d640743dd35da5931a7695a2d8121002e9ed33f99aeaae17ef018e689915f\": rpc error: code = NotFound desc = could not find container \"732d640743dd35da5931a7695a2d8121002e9ed33f99aeaae17ef018e689915f\": container with ID starting with 732d640743dd35da5931a7695a2d8121002e9ed33f99aeaae17ef018e689915f not found: ID does not exist"
Nov 02 02:28:54.276952 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:54.276867    5639 scope.go:122] "RemoveContainer" containerID="a7a48e157a60cac22f7ca18fc5dfde8ba6bc7081a40550723a671e2f6d6647ba"
Nov 02 02:28:54.285410 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:54.285392    5639 scope.go:122] "RemoveContainer" containerID="a7a48e157a60cac22f7ca18fc5dfde8ba6bc7081a40550723a671e2f6d6647ba"
Nov 02 02:28:54.285665 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:28:54.285645    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7a48e157a60cac22f7ca18fc5dfde8ba6bc7081a40550723a671e2f6d6647ba\": container with ID starting with a7a48e157a60cac22f7ca18fc5dfde8ba6bc7081a40550723a671e2f6d6647ba not found: ID does not exist" containerID="a7a48e157a60cac22f7ca18fc5dfde8ba6bc7081a40550723a671e2f6d6647ba"
Nov 02 02:28:54.285717 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:54.285673    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7a48e157a60cac22f7ca18fc5dfde8ba6bc7081a40550723a671e2f6d6647ba"} err="failed to get container status \"a7a48e157a60cac22f7ca18fc5dfde8ba6bc7081a40550723a671e2f6d6647ba\": rpc error: code = NotFound desc = could not find container \"a7a48e157a60cac22f7ca18fc5dfde8ba6bc7081a40550723a671e2f6d6647ba\": container with ID starting with a7a48e157a60cac22f7ca18fc5dfde8ba6bc7081a40550723a671e2f6d6647ba not found: ID does not exist"
Nov 02 02:28:54.431202 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:54.431170    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4"]
Nov 02 02:28:54.480784 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:54.480740    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ffe96-797ffscfg4"]
Nov 02 02:28:54.545368 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:54.545267    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd"]
Nov 02 02:28:54.550022 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:54.549996    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3ffe96-8cc7dd88f5jhwd"]
Nov 02 02:28:55.217293 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:55.217245    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2184c94a-607a-45ea-bf1f-fcba87b1259b" path="/var/lib/kubelet/pods/2184c94a-607a-45ea-bf1f-fcba87b1259b/volumes"
Nov 02 02:28:55.217683 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:55.217668    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc79cdac-eab8-4b50-9aa3-f2ce55132e0a" path="/var/lib/kubelet/pods/bc79cdac-eab8-4b50-9aa3-f2ce55132e0a/volumes"
Nov 02 02:28:56.370617 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:56.370561    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv" podUID="8a0a1833-f7b7-4cb2-932a-3006c4c66c1b" containerName="main" containerID="cri-o://4cf35d78433ce7ce38035e3e18789ff1c1c39f929f2764fceefcd684fdcfddb4" gracePeriod=45
Nov 02 02:28:56.627369 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:56.627305    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"
Nov 02 02:28:56.725841 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:56.725806    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-dshm\") pod \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\" (UID: \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\") "
Nov 02 02:28:56.726048 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:56.725856    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-tmp-dir\") pod \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\" (UID: \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\") "
Nov 02 02:28:56.726048 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:56.725901    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-model-cache\") pod \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\" (UID: \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\") "
Nov 02 02:28:56.726048 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:56.725951    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-home\" (UniqueName: \"kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-home\") pod \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\" (UID: \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\") "
Nov 02 02:28:56.726048 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:56.725995    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-tls-certs\") pod \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\" (UID: \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\") "
Nov 02 02:28:56.726048 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:56.726013    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-kube-api-access-7v2mx\" (UniqueName: \"kubernetes.io/projected/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-kube-api-access-7v2mx\") pod \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\" (UID: \"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b\") "
Nov 02 02:28:56.726401 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:56.726184    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-model-cache" pod "8a0a1833-f7b7-4cb2-932a-3006c4c66c1b" (UID: "8a0a1833-f7b7-4cb2-932a-3006c4c66c1b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:28:56.726401 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:56.726197    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-tmp-dir" pod "8a0a1833-f7b7-4cb2-932a-3006c4c66c1b" (UID: "8a0a1833-f7b7-4cb2-932a-3006c4c66c1b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:28:56.726401 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:56.726313    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-home" pod "8a0a1833-f7b7-4cb2-932a-3006c4c66c1b" (UID: "8a0a1833-f7b7-4cb2-932a-3006c4c66c1b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:28:56.726564 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:56.726461    5639 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-home\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:28:56.726564 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:56.726481    5639 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-tmp-dir\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:28:56.726564 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:56.726494    5639 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-model-cache\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:28:56.728375 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:56.728343    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-dshm" pod "8a0a1833-f7b7-4cb2-932a-3006c4c66c1b" (UID: "8a0a1833-f7b7-4cb2-932a-3006c4c66c1b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:28:56.728478 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:56.728416    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-tls-certs" pod "8a0a1833-f7b7-4cb2-932a-3006c4c66c1b" (UID: "8a0a1833-f7b7-4cb2-932a-3006c4c66c1b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:28:56.728580 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:56.728558    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-kube-api-access-7v2mx" pod "8a0a1833-f7b7-4cb2-932a-3006c4c66c1b" (UID: "8a0a1833-f7b7-4cb2-932a-3006c4c66c1b"). InnerVolumeSpecName "kube-api-access-7v2mx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:28:56.827729 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:56.827699    5639 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-tls-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:28:56.827729 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:56.827729    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v2mx\" (UniqueName: \"kubernetes.io/projected/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-kube-api-access-7v2mx\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:28:56.827933 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:56.827741    5639 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b-dshm\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:28:56.977902 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:56.977807    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5"]
Nov 02 02:28:56.980307 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:56.980267    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ffe96-jl4s5"]
Nov 02 02:28:57.047313 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:57.047233    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d57d4200-jlbtw"]
Nov 02 02:28:57.049617 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:57.049589    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d57d4200-jlbtw"]
Nov 02 02:28:57.217382 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:57.217351    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d229e732-b43a-45e8-8ea4-93302a4274b1" path="/var/lib/kubelet/pods/d229e732-b43a-45e8-8ea4-93302a4274b1/volumes"
Nov 02 02:28:57.217789 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:57.217776    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb2272f9-5d1c-4dc3-beaa-bdd278852801" path="/var/lib/kubelet/pods/eb2272f9-5d1c-4dc3-beaa-bdd278852801/volumes"
Nov 02 02:28:57.287157 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:57.287060    5639 generic.go:356] "Generic (PLEG): container finished" podID="8a0a1833-f7b7-4cb2-932a-3006c4c66c1b" containerID="4cf35d78433ce7ce38035e3e18789ff1c1c39f929f2764fceefcd684fdcfddb4" exitCode=0
Nov 02 02:28:57.287157 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:57.287109    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"
Nov 02 02:28:57.287428 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:57.287150    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv" event={"ID":"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b","Type":"ContainerDied","Data":"4cf35d78433ce7ce38035e3e18789ff1c1c39f929f2764fceefcd684fdcfddb4"}
Nov 02 02:28:57.287428 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:57.287192    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv" event={"ID":"8a0a1833-f7b7-4cb2-932a-3006c4c66c1b","Type":"ContainerDied","Data":"c6c16dee02f25ab6086297635edc6ba9d8a2b82da6070c846d0b2b6eff4c20d4"}
Nov 02 02:28:57.287428 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:57.287210    5639 scope.go:122] "RemoveContainer" containerID="4cf35d78433ce7ce38035e3e18789ff1c1c39f929f2764fceefcd684fdcfddb4"
Nov 02 02:28:57.296307 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:57.296271    5639 scope.go:122] "RemoveContainer" containerID="4cf35d78433ce7ce38035e3e18789ff1c1c39f929f2764fceefcd684fdcfddb4"
Nov 02 02:28:57.296603 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:28:57.296584    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf35d78433ce7ce38035e3e18789ff1c1c39f929f2764fceefcd684fdcfddb4\": container with ID starting with 4cf35d78433ce7ce38035e3e18789ff1c1c39f929f2764fceefcd684fdcfddb4 not found: ID does not exist" containerID="4cf35d78433ce7ce38035e3e18789ff1c1c39f929f2764fceefcd684fdcfddb4"
Nov 02 02:28:57.296691 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:57.296613    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf35d78433ce7ce38035e3e18789ff1c1c39f929f2764fceefcd684fdcfddb4"} err="failed to get container status \"4cf35d78433ce7ce38035e3e18789ff1c1c39f929f2764fceefcd684fdcfddb4\": rpc error: code = NotFound desc = could not find container \"4cf35d78433ce7ce38035e3e18789ff1c1c39f929f2764fceefcd684fdcfddb4\": container with ID starting with 4cf35d78433ce7ce38035e3e18789ff1c1c39f929f2764fceefcd684fdcfddb4 not found: ID does not exist"
Nov 02 02:28:57.419492 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:28:57.419450    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:28:57.450509 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:57.450474    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"]
Nov 02 02:28:57.454969 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:57.454943    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-3ffe96/simulator-single-main-3ffe96-kserve-66b4f7b8fc-5b7hv"]
Nov 02 02:28:59.219250 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:59.219218    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a0a1833-f7b7-4cb2-932a-3006c4c66c1b" path="/var/lib/kubelet/pods/8a0a1833-f7b7-4cb2-932a-3006c4c66c1b/volumes"
Nov 02 02:28:59.604673 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:28:59.604638    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:29:03.783663 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:03.783631    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"]
Nov 02 02:29:03.784050 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:03.783854    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv" podUID="89e9e3aa-6453-4ada-a8f4-f59470d5c9a9" containerName="istio-proxy" containerID="cri-o://2f905d524dd514268a3d8e23df9d75224608d671270c76dfc8196ff43506691f" gracePeriod=30
Nov 02 02:29:07.700085 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:07.700048    5639 patch_prober.go:29] interesting pod/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:29:07.700582 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:07.700117    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv" podUID="89e9e3aa-6453-4ada-a8f4-f59470d5c9a9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:29:09.045596 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.045568    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:29:09.151569 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.151538    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-workload-certs\") pod \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") "
Nov 02 02:29:09.151776 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.151577    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-podinfo\") pod \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") "
Nov 02 02:29:09.151776 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.151631    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-envoy\") pod \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") "
Nov 02 02:29:09.151776 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.151667    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istiod-ca-cert\") pod \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") "
Nov 02 02:29:09.151776 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.151699    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-kube-api-access-l2z8g\" (UniqueName: \"kubernetes.io/projected/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-kube-api-access-l2z8g\") pod \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") "
Nov 02 02:29:09.151776 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.151732    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-workload-socket\") pod \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") "
Nov 02 02:29:09.152073 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.151808    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-data\") pod \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") "
Nov 02 02:29:09.152073 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.151844    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-workload-certs" pod "89e9e3aa-6453-4ada-a8f4-f59470d5c9a9" (UID: "89e9e3aa-6453-4ada-a8f4-f59470d5c9a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:29:09.152073 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.151896    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-credential-socket\") pod \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") "
Nov 02 02:29:09.152073 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.151915    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-token\" (UniqueName: \"kubernetes.io/projected/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-token\") pod \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\" (UID: \"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9\") "
Nov 02 02:29:09.152312 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.152135    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-data" pod "89e9e3aa-6453-4ada-a8f4-f59470d5c9a9" (UID: "89e9e3aa-6453-4ada-a8f4-f59470d5c9a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:29:09.152312 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.152139    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-credential-socket" pod "89e9e3aa-6453-4ada-a8f4-f59470d5c9a9" (UID: "89e9e3aa-6453-4ada-a8f4-f59470d5c9a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:29:09.152312 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.152214    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-workload-socket" pod "89e9e3aa-6453-4ada-a8f4-f59470d5c9a9" (UID: "89e9e3aa-6453-4ada-a8f4-f59470d5c9a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:29:09.152472 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.152305    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istiod-ca-cert" pod "89e9e3aa-6453-4ada-a8f4-f59470d5c9a9" (UID: "89e9e3aa-6453-4ada-a8f4-f59470d5c9a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:29:09.152472 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.152363    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:29:09.152472 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.152383    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:29:09.152472 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.152397    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:29:09.152472 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.152409    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:29:09.154216 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.154179    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-podinfo" pod "89e9e3aa-6453-4ada-a8f4-f59470d5c9a9" (UID: "89e9e3aa-6453-4ada-a8f4-f59470d5c9a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:29:09.154619 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.154590    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-token" pod "89e9e3aa-6453-4ada-a8f4-f59470d5c9a9" (UID: "89e9e3aa-6453-4ada-a8f4-f59470d5c9a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:29:09.154809 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.154789    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-envoy" pod "89e9e3aa-6453-4ada-a8f4-f59470d5c9a9" (UID: "89e9e3aa-6453-4ada-a8f4-f59470d5c9a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:29:09.154868 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.154809    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-kube-api-access-l2z8g" pod "89e9e3aa-6453-4ada-a8f4-f59470d5c9a9" (UID: "89e9e3aa-6453-4ada-a8f4-f59470d5c9a9"). InnerVolumeSpecName "kube-api-access-l2z8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:29:09.253979 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.253945    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:29:09.253979 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.253970    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:29:09.253979 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.253980    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:29:09.254250 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.253993    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:29:09.254250 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.254007    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2z8g\" (UniqueName: \"kubernetes.io/projected/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9-kube-api-access-l2z8g\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:29:09.351392 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.351353    5639 generic.go:356] "Generic (PLEG): container finished" podID="89e9e3aa-6453-4ada-a8f4-f59470d5c9a9" containerID="2f905d524dd514268a3d8e23df9d75224608d671270c76dfc8196ff43506691f" exitCode=0
Nov 02 02:29:09.351560 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.351414    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"
Nov 02 02:29:09.351560 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.351435    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv" event={"ID":"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9","Type":"ContainerDied","Data":"2f905d524dd514268a3d8e23df9d75224608d671270c76dfc8196ff43506691f"}
Nov 02 02:29:09.351560 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.351496    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv" event={"ID":"89e9e3aa-6453-4ada-a8f4-f59470d5c9a9","Type":"ContainerDied","Data":"708e810dc762b031e41a29099cf06ba4a96f39904faace12026fd46c5209feaa"}
Nov 02 02:29:09.351560 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.351518    5639 scope.go:122] "RemoveContainer" containerID="2f905d524dd514268a3d8e23df9d75224608d671270c76dfc8196ff43506691f"
Nov 02 02:29:09.360513 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.360494    5639 scope.go:122] "RemoveContainer" containerID="636814bb41a1984322cd2fa6ae6228ec59077f726f39ccdbdb215ec1634d3e4d"
Nov 02 02:29:09.368715 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.368696    5639 scope.go:122] "RemoveContainer" containerID="2f905d524dd514268a3d8e23df9d75224608d671270c76dfc8196ff43506691f"
Nov 02 02:29:09.368983 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:29:09.368959    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f905d524dd514268a3d8e23df9d75224608d671270c76dfc8196ff43506691f\": container with ID starting with 2f905d524dd514268a3d8e23df9d75224608d671270c76dfc8196ff43506691f not found: ID does not exist" containerID="2f905d524dd514268a3d8e23df9d75224608d671270c76dfc8196ff43506691f"
Nov 02 02:29:09.369084 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.368992    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f905d524dd514268a3d8e23df9d75224608d671270c76dfc8196ff43506691f"} err="failed to get container status \"2f905d524dd514268a3d8e23df9d75224608d671270c76dfc8196ff43506691f\": rpc error: code = NotFound desc = could not find container \"2f905d524dd514268a3d8e23df9d75224608d671270c76dfc8196ff43506691f\": container with ID starting with 2f905d524dd514268a3d8e23df9d75224608d671270c76dfc8196ff43506691f not found: ID does not exist"
Nov 02 02:29:09.369084 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.369013    5639 scope.go:122] "RemoveContainer" containerID="636814bb41a1984322cd2fa6ae6228ec59077f726f39ccdbdb215ec1634d3e4d"
Nov 02 02:29:09.369273 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:29:09.369258    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"636814bb41a1984322cd2fa6ae6228ec59077f726f39ccdbdb215ec1634d3e4d\": container with ID starting with 636814bb41a1984322cd2fa6ae6228ec59077f726f39ccdbdb215ec1634d3e4d not found: ID does not exist" containerID="636814bb41a1984322cd2fa6ae6228ec59077f726f39ccdbdb215ec1634d3e4d"
Nov 02 02:29:09.369342 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.369295    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"636814bb41a1984322cd2fa6ae6228ec59077f726f39ccdbdb215ec1634d3e4d"} err="failed to get container status \"636814bb41a1984322cd2fa6ae6228ec59077f726f39ccdbdb215ec1634d3e4d\": rpc error: code = NotFound desc = could not find container \"636814bb41a1984322cd2fa6ae6228ec59077f726f39ccdbdb215ec1634d3e4d\": container with ID starting with 636814bb41a1984322cd2fa6ae6228ec59077f726f39ccdbdb215ec1634d3e4d not found: ID does not exist"
Nov 02 02:29:09.515234 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.515153    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"]
Nov 02 02:29:09.520543 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:09.520518    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3ffe96-openshift-default-979f6ddcc-4vxmv"]
Nov 02 02:29:10.048141 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.048103    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"]
Nov 02 02:29:10.048776 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.048732    5639 cpu_manager.go:414] "containerMap: removing container" podUID="89e9e3aa-6453-4ada-a8f4-f59470d5c9a9" containerName="istio-proxy"
Nov 02 02:29:10.048776 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.048755    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89e9e3aa-6453-4ada-a8f4-f59470d5c9a9" containerName="istio-proxy"
Nov 02 02:29:10.048776 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.048772    5639 cpu_manager.go:414] "containerMap: removing container" podUID="8a0a1833-f7b7-4cb2-932a-3006c4c66c1b" containerName="main"
Nov 02 02:29:10.048776 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.048779    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a0a1833-f7b7-4cb2-932a-3006c4c66c1b" containerName="main"
Nov 02 02:29:10.049038 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.048814    5639 cpu_manager.go:414] "containerMap: removing container" podUID="bc79cdac-eab8-4b50-9aa3-f2ce55132e0a" containerName="payload-processing"
Nov 02 02:29:10.049038 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.048822    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc79cdac-eab8-4b50-9aa3-f2ce55132e0a" containerName="payload-processing"
Nov 02 02:29:10.049038 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.048843    5639 cpu_manager.go:414] "containerMap: removing container" podUID="2184c94a-607a-45ea-bf1f-fcba87b1259b" containerName="payload-pre-processing"
Nov 02 02:29:10.049038 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.048851    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2184c94a-607a-45ea-bf1f-fcba87b1259b" containerName="payload-pre-processing"
Nov 02 02:29:10.049038 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.048862    5639 cpu_manager.go:414] "containerMap: removing container" podUID="89e9e3aa-6453-4ada-a8f4-f59470d5c9a9" containerName="istio-proxy"
Nov 02 02:29:10.049038 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.048869    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89e9e3aa-6453-4ada-a8f4-f59470d5c9a9" containerName="istio-proxy"
Nov 02 02:29:10.049038 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.048906    5639 cpu_manager.go:414] "containerMap: removing container" podUID="d229e732-b43a-45e8-8ea4-93302a4274b1" containerName="revoke-keys"
Nov 02 02:29:10.049038 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.048915    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d229e732-b43a-45e8-8ea4-93302a4274b1" containerName="revoke-keys"
Nov 02 02:29:10.049038 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.049021    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="d229e732-b43a-45e8-8ea4-93302a4274b1" containerName="revoke-keys"
Nov 02 02:29:10.049038 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.049033    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="89e9e3aa-6453-4ada-a8f4-f59470d5c9a9" containerName="istio-proxy"
Nov 02 02:29:10.049038 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.049041    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a0a1833-f7b7-4cb2-932a-3006c4c66c1b" containerName="main"
Nov 02 02:29:10.049624 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.049056    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc79cdac-eab8-4b50-9aa3-f2ce55132e0a" containerName="payload-processing"
Nov 02 02:29:10.049624 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.049066    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="2184c94a-607a-45ea-bf1f-fcba87b1259b" containerName="payload-pre-processing"
Nov 02 02:29:10.049624 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.049074    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="89e9e3aa-6453-4ada-a8f4-f59470d5c9a9" containerName="istio-proxy"
Nov 02 02:29:10.051840 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.051820    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"
Nov 02 02:29:10.104191 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.104166    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-eefc7478-kserve-self-signed-certs\""
Nov 02 02:29:10.155325 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.155273    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"]
Nov 02 02:29:10.163624 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.163592    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-model-cache\") pod \"e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb\" (UID: \"6bb83e70-1742-448d-abc9-a4dc65b6b970\") " pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"
Nov 02 02:29:10.163781 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.163649    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-646s2\" (UniqueName: \"kubernetes.io/projected/6bb83e70-1742-448d-abc9-a4dc65b6b970-kube-api-access-646s2\") pod \"e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb\" (UID: \"6bb83e70-1742-448d-abc9-a4dc65b6b970\") " pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"
Nov 02 02:29:10.163781 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.163673    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-tmp-dir\") pod \"e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb\" (UID: \"6bb83e70-1742-448d-abc9-a4dc65b6b970\") " pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"
Nov 02 02:29:10.163873 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.163777    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb83e70-1742-448d-abc9-a4dc65b6b970-tls-certs\") pod \"e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb\" (UID: \"6bb83e70-1742-448d-abc9-a4dc65b6b970\") " pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"
Nov 02 02:29:10.163873 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.163810    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-dshm\") pod \"e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb\" (UID: \"6bb83e70-1742-448d-abc9-a4dc65b6b970\") " pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"
Nov 02 02:29:10.163873 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.163827    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-home\") pod \"e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb\" (UID: \"6bb83e70-1742-448d-abc9-a4dc65b6b970\") " pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"
Nov 02 02:29:10.264786 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.264750    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-646s2\" (UniqueName: \"kubernetes.io/projected/6bb83e70-1742-448d-abc9-a4dc65b6b970-kube-api-access-646s2\") pod \"e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb\" (UID: \"6bb83e70-1742-448d-abc9-a4dc65b6b970\") " pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"
Nov 02 02:29:10.265003 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.264792    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-tmp-dir\") pod \"e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb\" (UID: \"6bb83e70-1742-448d-abc9-a4dc65b6b970\") " pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"
Nov 02 02:29:10.265003 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.264826    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb83e70-1742-448d-abc9-a4dc65b6b970-tls-certs\") pod \"e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb\" (UID: \"6bb83e70-1742-448d-abc9-a4dc65b6b970\") " pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"
Nov 02 02:29:10.265003 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.264850    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-dshm\") pod \"e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb\" (UID: \"6bb83e70-1742-448d-abc9-a4dc65b6b970\") " pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"
Nov 02 02:29:10.265003 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.264989    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-home\") pod \"e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb\" (UID: \"6bb83e70-1742-448d-abc9-a4dc65b6b970\") " pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"
Nov 02 02:29:10.265236 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.265133    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-model-cache\") pod \"e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb\" (UID: \"6bb83e70-1742-448d-abc9-a4dc65b6b970\") " pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"
Nov 02 02:29:10.265341 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.265307    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-tmp-dir\") pod \"e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb\" (UID: \"6bb83e70-1742-448d-abc9-a4dc65b6b970\") " pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"
Nov 02 02:29:10.265414 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.265385    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-home\") pod \"e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb\" (UID: \"6bb83e70-1742-448d-abc9-a4dc65b6b970\") " pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"
Nov 02 02:29:10.265507 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.265490    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-model-cache\") pod \"e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb\" (UID: \"6bb83e70-1742-448d-abc9-a4dc65b6b970\") " pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"
Nov 02 02:29:10.267321 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.267302    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-dshm\") pod \"e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb\" (UID: \"6bb83e70-1742-448d-abc9-a4dc65b6b970\") " pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"
Nov 02 02:29:10.267601 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.267583    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb83e70-1742-448d-abc9-a4dc65b6b970-tls-certs\") pod \"e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb\" (UID: \"6bb83e70-1742-448d-abc9-a4dc65b6b970\") " pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"
Nov 02 02:29:10.325645 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.325579    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-646s2\" (UniqueName: \"kubernetes.io/projected/6bb83e70-1742-448d-abc9-a4dc65b6b970-kube-api-access-646s2\") pod \"e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb\" (UID: \"6bb83e70-1742-448d-abc9-a4dc65b6b970\") " pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"
Nov 02 02:29:10.363254 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.363226    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"
Nov 02 02:29:10.739243 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:10.739219    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"]
Nov 02 02:29:10.741603 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:29:10.741574    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bb83e70_1742_448d_abc9_a4dc65b6b970.slice/crio-f3ba905d14ac1c15142a5c255cab4709419992996541f1ff1f3ee7014c3bc111 WatchSource:0}: Error finding container f3ba905d14ac1c15142a5c255cab4709419992996541f1ff1f3ee7014c3bc111: Status 404 returned error can't find the container with id f3ba905d14ac1c15142a5c255cab4709419992996541f1ff1f3ee7014c3bc111
Nov 02 02:29:11.217167 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:11.217135    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89e9e3aa-6453-4ada-a8f4-f59470d5c9a9" path="/var/lib/kubelet/pods/89e9e3aa-6453-4ada-a8f4-f59470d5c9a9/volumes"
Nov 02 02:29:11.364466 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:11.364430    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb" event={"ID":"6bb83e70-1742-448d-abc9-a4dc65b6b970","Type":"ContainerStarted","Data":"bc4fa6516ad6f38e30d6b7c7afc56f74135e7d0e95ea8591459e22be3f13aef7"}
Nov 02 02:29:11.364466 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:11.364470    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb" event={"ID":"6bb83e70-1742-448d-abc9-a4dc65b6b970","Type":"ContainerStarted","Data":"f3ba905d14ac1c15142a5c255cab4709419992996541f1ff1f3ee7014c3bc111"}
Nov 02 02:29:11.481725 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:11.481624    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb" podStartSLOduration=1.4816109800000001 podStartE2EDuration="1.48161098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:29:11.480486178 +0000 UTC m=+3816.384168293" watchObservedRunningTime="2026-11-02 02:29:11.48161098 +0000 UTC m=+3816.385293094"
Nov 02 02:29:12.364000 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:12.363966    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"
Nov 02 02:29:12.804352 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:12.804251    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:29:20.363820 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:20.363783    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"
Nov 02 02:29:20.376941 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:20.376917    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"
Nov 02 02:29:20.477501 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:20.477470    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"
Nov 02 02:29:27.419319 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:29:27.419262    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:29:36.051546 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:36.051511    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9swvr"]
Nov 02 02:29:36.100616 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:36.100583    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9swvr"]
Nov 02 02:29:37.217597 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:37.217563    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8866053-37ae-4643-a307-95b0542ac721" path="/var/lib/kubelet/pods/e8866053-37ae-4643-a307-95b0542ac721/volumes"
Nov 02 02:29:51.935467 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:51.935431    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"]
Nov 02 02:29:51.939107 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:51.939087    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"
Nov 02 02:29:52.030274 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:52.030235    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-home\") pod \"e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m\" (UID: \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\") " pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"
Nov 02 02:29:52.030475 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:52.030306    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-tmp-dir\") pod \"e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m\" (UID: \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\") " pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"
Nov 02 02:29:52.030475 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:52.030464    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-dshm\") pod \"e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m\" (UID: \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\") " pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"
Nov 02 02:29:52.030590 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:52.030494    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-model-cache\") pod \"e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m\" (UID: \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\") " pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"
Nov 02 02:29:52.030590 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:52.030527    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b43fa6c-393b-4f14-bc93-ee58fae93cce-tls-certs\") pod \"e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m\" (UID: \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\") " pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"
Nov 02 02:29:52.030590 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:52.030574    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpzd9\" (UniqueName: \"kubernetes.io/projected/6b43fa6c-393b-4f14-bc93-ee58fae93cce-kube-api-access-xpzd9\") pod \"e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m\" (UID: \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\") " pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"
Nov 02 02:29:52.040975 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:52.040951    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-eefc7478-kserve-self-signed-certs\""
Nov 02 02:29:52.043038 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:52.043016    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"]
Nov 02 02:29:52.131670 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:52.131626    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-model-cache\") pod \"e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m\" (UID: \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\") " pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"
Nov 02 02:29:52.131883 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:52.131687    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b43fa6c-393b-4f14-bc93-ee58fae93cce-tls-certs\") pod \"e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m\" (UID: \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\") " pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"
Nov 02 02:29:52.131883 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:52.131817    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpzd9\" (UniqueName: \"kubernetes.io/projected/6b43fa6c-393b-4f14-bc93-ee58fae93cce-kube-api-access-xpzd9\") pod \"e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m\" (UID: \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\") " pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"
Nov 02 02:29:52.132030 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:52.131935    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-home\") pod \"e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m\" (UID: \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\") " pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"
Nov 02 02:29:52.132030 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:52.131958    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-tmp-dir\") pod \"e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m\" (UID: \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\") " pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"
Nov 02 02:29:52.132161 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:52.132051    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-dshm\") pod \"e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m\" (UID: \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\") " pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"
Nov 02 02:29:52.132161 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:52.132112    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-model-cache\") pod \"e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m\" (UID: \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\") " pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"
Nov 02 02:29:52.132372 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:52.132339    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-home\") pod \"e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m\" (UID: \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\") " pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"
Nov 02 02:29:52.132512 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:52.132478    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-tmp-dir\") pod \"e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m\" (UID: \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\") " pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"
Nov 02 02:29:52.134496 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:52.134474    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-dshm\") pod \"e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m\" (UID: \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\") " pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"
Nov 02 02:29:52.134577 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:52.134534    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b43fa6c-393b-4f14-bc93-ee58fae93cce-tls-certs\") pod \"e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m\" (UID: \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\") " pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"
Nov 02 02:29:52.196303 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:52.196198    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpzd9\" (UniqueName: \"kubernetes.io/projected/6b43fa6c-393b-4f14-bc93-ee58fae93cce-kube-api-access-xpzd9\") pod \"e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m\" (UID: \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\") " pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"
Nov 02 02:29:52.251401 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:52.251361    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"
Nov 02 02:29:52.636821 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:52.636791    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"]
Nov 02 02:29:52.639752 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:29:52.639722    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b43fa6c_393b_4f14_bc93_ee58fae93cce.slice/crio-95e8dd85a910a9ae77858e3e0a1e1c468d13919bbbb771d9c5cc06df35146ac6 WatchSource:0}: Error finding container 95e8dd85a910a9ae77858e3e0a1e1c468d13919bbbb771d9c5cc06df35146ac6: Status 404 returned error can't find the container with id 95e8dd85a910a9ae77858e3e0a1e1c468d13919bbbb771d9c5cc06df35146ac6
Nov 02 02:29:53.647090 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:53.647054    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m" event={"ID":"6b43fa6c-393b-4f14-bc93-ee58fae93cce","Type":"ContainerStarted","Data":"80a2dc24b786be1e587d8d69f9d1c7e2b865cb928c830f49138bcc3c56ff91fa"}
Nov 02 02:29:53.647090 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:53.647095    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m" event={"ID":"6b43fa6c-393b-4f14-bc93-ee58fae93cce","Type":"ContainerStarted","Data":"95e8dd85a910a9ae77858e3e0a1e1c468d13919bbbb771d9c5cc06df35146ac6"}
Nov 02 02:29:53.765353 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:53.765301    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m" podStartSLOduration=2.765269672 podStartE2EDuration="2.765269672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:29:53.764036806 +0000 UTC m=+3858.667718962" watchObservedRunningTime="2026-11-02 02:29:53.765269672 +0000 UTC m=+3858.668951776"
Nov 02 02:29:54.252438 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:54.252398    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"
Nov 02 02:29:56.597522 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:29:56.597472    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:29:57.426170 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:29:57.426108    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:30:00.168308 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:00.168200    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c"]
Nov 02 02:30:00.177374 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:00.177334    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c"
Nov 02 02:30:00.236693 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:00.236653    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c"]
Nov 02 02:30:00.257145 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:00.257107    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e096a67b-5970-446b-999c-d58a0c6a9cab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893110-b5g6c\" (UID: \"e096a67b-5970-446b-999c-d58a0c6a9cab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c"
Nov 02 02:30:00.257492 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:00.257450    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n8fb\" (UniqueName: \"kubernetes.io/projected/e096a67b-5970-446b-999c-d58a0c6a9cab-kube-api-access-8n8fb\") pod \"maas-api-key-cleanup-29893110-b5g6c\" (UID: \"e096a67b-5970-446b-999c-d58a0c6a9cab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c"
Nov 02 02:30:00.333749 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:00.333702    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:30:00.334992 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:00.334966    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:30:00.335328 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:00.335312    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:30:00.442823 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:00.442733    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e096a67b-5970-446b-999c-d58a0c6a9cab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893110-b5g6c\" (UID: \"e096a67b-5970-446b-999c-d58a0c6a9cab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c"
Nov 02 02:30:00.443947 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:00.443771    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e096a67b-5970-446b-999c-d58a0c6a9cab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893110-b5g6c\" (UID: \"e096a67b-5970-446b-999c-d58a0c6a9cab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c"
Nov 02 02:30:00.444295 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:00.444138    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n8fb\" (UniqueName: \"kubernetes.io/projected/e096a67b-5970-446b-999c-d58a0c6a9cab-kube-api-access-8n8fb\") pod \"maas-api-key-cleanup-29893110-b5g6c\" (UID: \"e096a67b-5970-446b-999c-d58a0c6a9cab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c"
Nov 02 02:30:00.534257 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:00.534117    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n8fb\" (UniqueName: \"kubernetes.io/projected/e096a67b-5970-446b-999c-d58a0c6a9cab-kube-api-access-8n8fb\") pod \"maas-api-key-cleanup-29893110-b5g6c\" (UID: \"e096a67b-5970-446b-999c-d58a0c6a9cab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c"
Nov 02 02:30:00.817869 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:00.817786    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c"
Nov 02 02:30:01.190769 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:01.190748    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c"]
Nov 02 02:30:01.193316 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:30:01.193269    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode096a67b_5970_446b_999c_d58a0c6a9cab.slice/crio-60c204d33acdf76d45f0921423c4c3a17af30ddf0ea114bf4c629c6e88130878 WatchSource:0}: Error finding container 60c204d33acdf76d45f0921423c4c3a17af30ddf0ea114bf4c629c6e88130878: Status 404 returned error can't find the container with id 60c204d33acdf76d45f0921423c4c3a17af30ddf0ea114bf4c629c6e88130878
Nov 02 02:30:01.714557 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:01.714515    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c" event={"ID":"e096a67b-5970-446b-999c-d58a0c6a9cab","Type":"ContainerStarted","Data":"771148ed3faa838e197c67c43c9a6c44dd57b8f4684d052f406e94197381c515"}
Nov 02 02:30:01.714742 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:01.714582    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c" event={"ID":"e096a67b-5970-446b-999c-d58a0c6a9cab","Type":"ContainerStarted","Data":"60c204d33acdf76d45f0921423c4c3a17af30ddf0ea114bf4c629c6e88130878"}
Nov 02 02:30:01.828726 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:01.828671    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c" podStartSLOduration=1.828657223 podStartE2EDuration="1.828657223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:30:01.828468268 +0000 UTC m=+3866.732150382" watchObservedRunningTime="2026-11-02 02:30:01.828657223 +0000 UTC m=+3866.732339336"
Nov 02 02:30:02.251571 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:02.251472    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"
Nov 02 02:30:02.264701 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:02.264662    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"
Nov 02 02:30:02.721267 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:02.721227    5639 generic.go:356] "Generic (PLEG): container finished" podID="e096a67b-5970-446b-999c-d58a0c6a9cab" containerID="771148ed3faa838e197c67c43c9a6c44dd57b8f4684d052f406e94197381c515" exitCode=60
Nov 02 02:30:02.721509 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:02.721327    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c" event={"ID":"e096a67b-5970-446b-999c-d58a0c6a9cab","Type":"ContainerDied","Data":"771148ed3faa838e197c67c43c9a6c44dd57b8f4684d052f406e94197381c515"}
Nov 02 02:30:02.721612 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:02.721598    5639 scope.go:122] "RemoveContainer" containerID="771148ed3faa838e197c67c43c9a6c44dd57b8f4684d052f406e94197381c515"
Nov 02 02:30:02.733355 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:02.733319    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"
Nov 02 02:30:03.729996 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:03.729898    5639 generic.go:356] "Generic (PLEG): container finished" podID="e096a67b-5970-446b-999c-d58a0c6a9cab" containerID="6fb8e757129035db434b09cb4aa7d747bedb68851a4f344fb9fad39dbc716d85" exitCode=60
Nov 02 02:30:03.729996 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:03.729979    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c" event={"ID":"e096a67b-5970-446b-999c-d58a0c6a9cab","Type":"ContainerDied","Data":"6fb8e757129035db434b09cb4aa7d747bedb68851a4f344fb9fad39dbc716d85"}
Nov 02 02:30:03.730586 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:03.730047    5639 scope.go:122] "RemoveContainer" containerID="771148ed3faa838e197c67c43c9a6c44dd57b8f4684d052f406e94197381c515"
Nov 02 02:30:03.730586 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:03.730414    5639 scope.go:122] "RemoveContainer" containerID="6fb8e757129035db434b09cb4aa7d747bedb68851a4f344fb9fad39dbc716d85"
Nov 02 02:30:03.730737 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:30:03.730710    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893110-b5g6c_odh-ai-gateway-infra(e096a67b-5970-446b-999c-d58a0c6a9cab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c" podUID="e096a67b-5970-446b-999c-d58a0c6a9cab"
Nov 02 02:30:04.736811 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:04.736787    5639 scope.go:122] "RemoveContainer" containerID="6fb8e757129035db434b09cb4aa7d747bedb68851a4f344fb9fad39dbc716d85"
Nov 02 02:30:04.737209 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:30:04.737040    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893110-b5g6c_odh-ai-gateway-infra(e096a67b-5970-446b-999c-d58a0c6a9cab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c" podUID="e096a67b-5970-446b-999c-d58a0c6a9cab"
Nov 02 02:30:14.211548 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:14.211511    5639 scope.go:122] "RemoveContainer" containerID="6fb8e757129035db434b09cb4aa7d747bedb68851a4f344fb9fad39dbc716d85"
Nov 02 02:30:14.791234 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:14.791136    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c" event={"ID":"e096a67b-5970-446b-999c-d58a0c6a9cab","Type":"ContainerStarted","Data":"93e70a781ac981e239de9639efbe226a453406a7f14274e821d0cce7ceb5230c"}
Nov 02 02:30:15.798484 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:15.798400    5639 generic.go:356] "Generic (PLEG): container finished" podID="e096a67b-5970-446b-999c-d58a0c6a9cab" containerID="93e70a781ac981e239de9639efbe226a453406a7f14274e821d0cce7ceb5230c" exitCode=60
Nov 02 02:30:15.798971 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:15.798482    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c" event={"ID":"e096a67b-5970-446b-999c-d58a0c6a9cab","Type":"ContainerDied","Data":"93e70a781ac981e239de9639efbe226a453406a7f14274e821d0cce7ceb5230c"}
Nov 02 02:30:15.798971 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:15.798556    5639 scope.go:122] "RemoveContainer" containerID="6fb8e757129035db434b09cb4aa7d747bedb68851a4f344fb9fad39dbc716d85"
Nov 02 02:30:15.798971 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:15.798891    5639 scope.go:122] "RemoveContainer" containerID="93e70a781ac981e239de9639efbe226a453406a7f14274e821d0cce7ceb5230c"
Nov 02 02:30:15.799185 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:30:15.799135    5639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893110-b5g6c_odh-ai-gateway-infra(e096a67b-5970-446b-999c-d58a0c6a9cab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c" podUID="e096a67b-5970-446b-999c-d58a0c6a9cab"
Nov 02 02:30:15.925942 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:15.925900    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c"]
Nov 02 02:30:16.965521 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:16.965490    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c"
Nov 02 02:30:17.130165 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:17.130122    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e096a67b-5970-446b-999c-d58a0c6a9cab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e096a67b-5970-446b-999c-d58a0c6a9cab-maas-api-service-ca\") pod \"e096a67b-5970-446b-999c-d58a0c6a9cab\" (UID: \"e096a67b-5970-446b-999c-d58a0c6a9cab\") "
Nov 02 02:30:17.130440 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:17.130216    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e096a67b-5970-446b-999c-d58a0c6a9cab-kube-api-access-8n8fb\" (UniqueName: \"kubernetes.io/projected/e096a67b-5970-446b-999c-d58a0c6a9cab-kube-api-access-8n8fb\") pod \"e096a67b-5970-446b-999c-d58a0c6a9cab\" (UID: \"e096a67b-5970-446b-999c-d58a0c6a9cab\") "
Nov 02 02:30:17.130595 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:17.130568    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e096a67b-5970-446b-999c-d58a0c6a9cab-maas-api-service-ca" pod "e096a67b-5970-446b-999c-d58a0c6a9cab" (UID: "e096a67b-5970-446b-999c-d58a0c6a9cab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:30:17.130675 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:17.130652    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e096a67b-5970-446b-999c-d58a0c6a9cab-maas-api-service-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:30:17.132501 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:17.132461    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e096a67b-5970-446b-999c-d58a0c6a9cab-kube-api-access-8n8fb" pod "e096a67b-5970-446b-999c-d58a0c6a9cab" (UID: "e096a67b-5970-446b-999c-d58a0c6a9cab"). InnerVolumeSpecName "kube-api-access-8n8fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:30:17.231804 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:17.231776    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n8fb\" (UniqueName: \"kubernetes.io/projected/e096a67b-5970-446b-999c-d58a0c6a9cab-kube-api-access-8n8fb\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:30:17.812046 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:17.812015    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c" event={"ID":"e096a67b-5970-446b-999c-d58a0c6a9cab","Type":"ContainerDied","Data":"60c204d33acdf76d45f0921423c4c3a17af30ddf0ea114bf4c629c6e88130878"}
Nov 02 02:30:17.812249 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:17.812077    5639 scope.go:122] "RemoveContainer" containerID="93e70a781ac981e239de9639efbe226a453406a7f14274e821d0cce7ceb5230c"
Nov 02 02:30:17.812249 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:17.812022    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c"
Nov 02 02:30:17.979491 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:17.979462    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c"]
Nov 02 02:30:18.028908 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:18.028872    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893110-b5g6c"]
Nov 02 02:30:19.219181 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:19.219136    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e096a67b-5970-446b-999c-d58a0c6a9cab" path="/var/lib/kubelet/pods/e096a67b-5970-446b-999c-d58a0c6a9cab/volumes"
Nov 02 02:30:22.544362 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:22.544323    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5rtbc"]
Nov 02 02:30:22.544822 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:22.544753    5639 cpu_manager.go:414] "containerMap: removing container" podUID="e096a67b-5970-446b-999c-d58a0c6a9cab" containerName="cleanup"
Nov 02 02:30:22.544822 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:22.544764    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e096a67b-5970-446b-999c-d58a0c6a9cab" containerName="cleanup"
Nov 02 02:30:22.544822 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:22.544796    5639 cpu_manager.go:414] "containerMap: removing container" podUID="e096a67b-5970-446b-999c-d58a0c6a9cab" containerName="cleanup"
Nov 02 02:30:22.544822 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:22.544801    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e096a67b-5970-446b-999c-d58a0c6a9cab" containerName="cleanup"
Nov 02 02:30:22.545004 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:22.544866    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="e096a67b-5970-446b-999c-d58a0c6a9cab" containerName="cleanup"
Nov 02 02:30:22.545004 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:22.544881    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="e096a67b-5970-446b-999c-d58a0c6a9cab" containerName="cleanup"
Nov 02 02:30:22.545004 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:22.544887    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="e096a67b-5970-446b-999c-d58a0c6a9cab" containerName="cleanup"
Nov 02 02:30:22.549413 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:22.549394    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5rtbc"
Nov 02 02:30:22.602715 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:22.602672    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:30:22.652516 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:22.652474    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:30:22.652516 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:22.652495    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:30:22.652740 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:22.652541    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:30:22.654308 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:22.654271    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5rtbc"]
Nov 02 02:30:22.692815 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:22.692766    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb6129b9-e5b7-412a-a3f6-d7e8694db3f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-5rtbc\" (UID: \"cb6129b9-e5b7-412a-a3f6-d7e8694db3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5rtbc"
Nov 02 02:30:22.692991 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:22.692831    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb6129b9-e5b7-412a-a3f6-d7e8694db3f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-5rtbc\" (UID: \"cb6129b9-e5b7-412a-a3f6-d7e8694db3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5rtbc"
Nov 02 02:30:22.692991 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:22.692916    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbzkh\" (UniqueName: \"kubernetes.io/projected/cb6129b9-e5b7-412a-a3f6-d7e8694db3f7-kube-api-access-gbzkh\") pod \"maas-api-revoke-sub-452b6d7b-5rtbc\" (UID: \"cb6129b9-e5b7-412a-a3f6-d7e8694db3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5rtbc"
Nov 02 02:30:22.793970 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:22.793935    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb6129b9-e5b7-412a-a3f6-d7e8694db3f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-5rtbc\" (UID: \"cb6129b9-e5b7-412a-a3f6-d7e8694db3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5rtbc"
Nov 02 02:30:22.794160 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:22.793987    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb6129b9-e5b7-412a-a3f6-d7e8694db3f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-5rtbc\" (UID: \"cb6129b9-e5b7-412a-a3f6-d7e8694db3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5rtbc"
Nov 02 02:30:22.794160 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:22.794025    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbzkh\" (UniqueName: \"kubernetes.io/projected/cb6129b9-e5b7-412a-a3f6-d7e8694db3f7-kube-api-access-gbzkh\") pod \"maas-api-revoke-sub-452b6d7b-5rtbc\" (UID: \"cb6129b9-e5b7-412a-a3f6-d7e8694db3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5rtbc"
Nov 02 02:30:22.794618 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:22.794567    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb6129b9-e5b7-412a-a3f6-d7e8694db3f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-5rtbc\" (UID: \"cb6129b9-e5b7-412a-a3f6-d7e8694db3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5rtbc"
Nov 02 02:30:22.796495 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:22.796478    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb6129b9-e5b7-412a-a3f6-d7e8694db3f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-5rtbc\" (UID: \"cb6129b9-e5b7-412a-a3f6-d7e8694db3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5rtbc"
Nov 02 02:30:22.855100 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:22.855067    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbzkh\" (UniqueName: \"kubernetes.io/projected/cb6129b9-e5b7-412a-a3f6-d7e8694db3f7-kube-api-access-gbzkh\") pod \"maas-api-revoke-sub-452b6d7b-5rtbc\" (UID: \"cb6129b9-e5b7-412a-a3f6-d7e8694db3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5rtbc"
Nov 02 02:30:22.862809 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:22.862778    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5rtbc"
Nov 02 02:30:23.235263 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:23.235031    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5rtbc"]
Nov 02 02:30:23.237825 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:30:23.237795    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb6129b9_e5b7_412a_a3f6_d7e8694db3f7.slice/crio-1bf4d423a5f271f320e6d055b52c6378cd12538831c03ed3c1d372e8ee1001c2 WatchSource:0}: Error finding container 1bf4d423a5f271f320e6d055b52c6378cd12538831c03ed3c1d372e8ee1001c2: Status 404 returned error can't find the container with id 1bf4d423a5f271f320e6d055b52c6378cd12538831c03ed3c1d372e8ee1001c2
Nov 02 02:30:23.847818 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:23.847776    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5rtbc" event={"ID":"cb6129b9-e5b7-412a-a3f6-d7e8694db3f7","Type":"ContainerStarted","Data":"bc3b5cfbcad6d58f519af33764b1562b36d341ccb0f317071d766671a24d5d22"}
Nov 02 02:30:23.848327 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:23.847884    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5rtbc" event={"ID":"cb6129b9-e5b7-412a-a3f6-d7e8694db3f7","Type":"ContainerStarted","Data":"1bf4d423a5f271f320e6d055b52c6378cd12538831c03ed3c1d372e8ee1001c2"}
Nov 02 02:30:23.893832 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:23.893792    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:30:23.963533 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:23.963472    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5rtbc" podStartSLOduration=1.963454296 podStartE2EDuration="1.963454296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:30:23.962651919 +0000 UTC m=+3888.866334073" watchObservedRunningTime="2026-11-02 02:30:23.963454296 +0000 UTC m=+3888.867136412"
Nov 02 02:30:24.858243 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:24.858201    5639 generic.go:356] "Generic (PLEG): container finished" podID="cb6129b9-e5b7-412a-a3f6-d7e8694db3f7" containerID="bc3b5cfbcad6d58f519af33764b1562b36d341ccb0f317071d766671a24d5d22" exitCode=0
Nov 02 02:30:24.859051 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:24.858693    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5rtbc" event={"ID":"cb6129b9-e5b7-412a-a3f6-d7e8694db3f7","Type":"ContainerDied","Data":"bc3b5cfbcad6d58f519af33764b1562b36d341ccb0f317071d766671a24d5d22"}
Nov 02 02:30:26.841844 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:26.841512    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5rtbc"
Nov 02 02:30:26.927636 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:26.927539    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5rtbc" event={"ID":"cb6129b9-e5b7-412a-a3f6-d7e8694db3f7","Type":"ContainerDied","Data":"1bf4d423a5f271f320e6d055b52c6378cd12538831c03ed3c1d372e8ee1001c2"}
Nov 02 02:30:26.927915 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:26.927892    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bf4d423a5f271f320e6d055b52c6378cd12538831c03ed3c1d372e8ee1001c2"
Nov 02 02:30:26.928117 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:26.928100    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5rtbc"
Nov 02 02:30:27.040311 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:27.034773    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb6129b9-e5b7-412a-a3f6-d7e8694db3f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb6129b9-e5b7-412a-a3f6-d7e8694db3f7-maas-api-xks-ca\") pod \"cb6129b9-e5b7-412a-a3f6-d7e8694db3f7\" (UID: \"cb6129b9-e5b7-412a-a3f6-d7e8694db3f7\") "
Nov 02 02:30:27.040311 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:27.034854    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb6129b9-e5b7-412a-a3f6-d7e8694db3f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb6129b9-e5b7-412a-a3f6-d7e8694db3f7-maas-api-openshift-ca\") pod \"cb6129b9-e5b7-412a-a3f6-d7e8694db3f7\" (UID: \"cb6129b9-e5b7-412a-a3f6-d7e8694db3f7\") "
Nov 02 02:30:27.040311 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:27.034957    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb6129b9-e5b7-412a-a3f6-d7e8694db3f7-kube-api-access-gbzkh\" (UniqueName: \"kubernetes.io/projected/cb6129b9-e5b7-412a-a3f6-d7e8694db3f7-kube-api-access-gbzkh\") pod \"cb6129b9-e5b7-412a-a3f6-d7e8694db3f7\" (UID: \"cb6129b9-e5b7-412a-a3f6-d7e8694db3f7\") "
Nov 02 02:30:27.060956 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:27.060857    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb6129b9-e5b7-412a-a3f6-d7e8694db3f7-maas-api-openshift-ca" pod "cb6129b9-e5b7-412a-a3f6-d7e8694db3f7" (UID: "cb6129b9-e5b7-412a-a3f6-d7e8694db3f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:30:27.148377 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:27.148066    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb6129b9-e5b7-412a-a3f6-d7e8694db3f7-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:30:27.185569 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:27.185457    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb6129b9-e5b7-412a-a3f6-d7e8694db3f7-kube-api-access-gbzkh" pod "cb6129b9-e5b7-412a-a3f6-d7e8694db3f7" (UID: "cb6129b9-e5b7-412a-a3f6-d7e8694db3f7"). InnerVolumeSpecName "kube-api-access-gbzkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:30:27.197476 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:27.197417    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb6129b9-e5b7-412a-a3f6-d7e8694db3f7-maas-api-xks-ca" pod "cb6129b9-e5b7-412a-a3f6-d7e8694db3f7" (UID: "cb6129b9-e5b7-412a-a3f6-d7e8694db3f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:30:27.256189 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:27.256151    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbzkh\" (UniqueName: \"kubernetes.io/projected/cb6129b9-e5b7-412a-a3f6-d7e8694db3f7-kube-api-access-gbzkh\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:30:27.256490 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:27.256469    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb6129b9-e5b7-412a-a3f6-d7e8694db3f7-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:30:27.506675 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:30:27.506576    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:30:35.383554 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:35.383522    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"]
Nov 02 02:30:37.476792 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:37.476757    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:30:39.794040 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:39.793995    5639 scope.go:122] "RemoveContainer" containerID="708910316a1f666a162d13f9e5b7c298076ffcd5e61fb6de50db9179fb2b2ff5"
Nov 02 02:30:39.888615 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:39.884821    5639 scope.go:122] "RemoveContainer" containerID="21cb8e4eef9d162d3733cc240f50268e512fbd7ed37a5ec81ca15a2cd3c67338"
Nov 02 02:30:39.925499 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:39.925457    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-whtg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:30:39.926875 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:39.926836    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whtg2" podUID="9e6c6d59-73b1-4b69-94d5-586b38902af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:30:39.993515 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:39.993478    5639 scope.go:122] "RemoveContainer" containerID="49700903dcaea47328e3f0b88aa57770385a6356569239feea5b9a20df5bc44c"
Nov 02 02:30:40.371307 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:40.370035    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fd65x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:30:40.371307 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:40.370112    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x" podUID="faf2916d-803b-4f14-a2b1-1c45b742fc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:30:42.144306 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:42.130062    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"]
Nov 02 02:30:46.352273 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:46.352237    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-dn28l"]
Nov 02 02:30:46.352852 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:46.352723    5639 cpu_manager.go:414] "containerMap: removing container" podUID="e096a67b-5970-446b-999c-d58a0c6a9cab" containerName="cleanup"
Nov 02 02:30:46.352852 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:46.352735    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e096a67b-5970-446b-999c-d58a0c6a9cab" containerName="cleanup"
Nov 02 02:30:46.352852 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:46.352747    5639 cpu_manager.go:414] "containerMap: removing container" podUID="cb6129b9-e5b7-412a-a3f6-d7e8694db3f7" containerName="revoke-keys"
Nov 02 02:30:46.352852 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:46.352752    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb6129b9-e5b7-412a-a3f6-d7e8694db3f7" containerName="revoke-keys"
Nov 02 02:30:46.352852 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:46.352838    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb6129b9-e5b7-412a-a3f6-d7e8694db3f7" containerName="revoke-keys"
Nov 02 02:30:46.355930 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:46.355912    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-dn28l"
Nov 02 02:30:46.408684 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:46.408645    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:30:46.408684 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:46.408668    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:30:46.408684 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:46.408675    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:30:46.416168 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:46.416141    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa7252e-e373-47b5-8664-0931881b390a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-dn28l\" (UID: \"5aa7252e-e373-47b5-8664-0931881b390a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-dn28l"
Nov 02 02:30:46.416366 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:46.416198    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa7252e-e373-47b5-8664-0931881b390a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-dn28l\" (UID: \"5aa7252e-e373-47b5-8664-0931881b390a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-dn28l"
Nov 02 02:30:46.416366 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:46.416347    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47x45\" (UniqueName: \"kubernetes.io/projected/5aa7252e-e373-47b5-8664-0931881b390a-kube-api-access-47x45\") pod \"maas-api-revoke-sub-3e31a506-dn28l\" (UID: \"5aa7252e-e373-47b5-8664-0931881b390a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-dn28l"
Nov 02 02:30:46.458061 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:46.458031    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:30:46.460265 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:46.460241    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-dn28l"]
Nov 02 02:30:46.517500 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:46.517468    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa7252e-e373-47b5-8664-0931881b390a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-dn28l\" (UID: \"5aa7252e-e373-47b5-8664-0931881b390a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-dn28l"
Nov 02 02:30:46.517718 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:46.517554    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47x45\" (UniqueName: \"kubernetes.io/projected/5aa7252e-e373-47b5-8664-0931881b390a-kube-api-access-47x45\") pod \"maas-api-revoke-sub-3e31a506-dn28l\" (UID: \"5aa7252e-e373-47b5-8664-0931881b390a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-dn28l"
Nov 02 02:30:46.517718 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:46.517611    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa7252e-e373-47b5-8664-0931881b390a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-dn28l\" (UID: \"5aa7252e-e373-47b5-8664-0931881b390a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-dn28l"
Nov 02 02:30:46.518243 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:46.518220    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa7252e-e373-47b5-8664-0931881b390a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-dn28l\" (UID: \"5aa7252e-e373-47b5-8664-0931881b390a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-dn28l"
Nov 02 02:30:46.520039 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:46.520014    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa7252e-e373-47b5-8664-0931881b390a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-dn28l\" (UID: \"5aa7252e-e373-47b5-8664-0931881b390a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-dn28l"
Nov 02 02:30:46.579396 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:46.579355    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47x45\" (UniqueName: \"kubernetes.io/projected/5aa7252e-e373-47b5-8664-0931881b390a-kube-api-access-47x45\") pod \"maas-api-revoke-sub-3e31a506-dn28l\" (UID: \"5aa7252e-e373-47b5-8664-0931881b390a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-dn28l"
Nov 02 02:30:46.669711 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:46.669612    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-dn28l"
Nov 02 02:30:47.085732 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:47.085695    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-dn28l"]
Nov 02 02:30:47.091422 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:30:47.090654    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa7252e_e373_47b5_8664_0931881b390a.slice/crio-3684391e657aa22e0107a899d7b2741461bd9accb3494a8a275f18f9b929ba43 WatchSource:0}: Error finding container 3684391e657aa22e0107a899d7b2741461bd9accb3494a8a275f18f9b929ba43: Status 404 returned error can't find the container with id 3684391e657aa22e0107a899d7b2741461bd9accb3494a8a275f18f9b929ba43
Nov 02 02:30:47.166443 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:47.166386    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-dn28l" event={"ID":"5aa7252e-e373-47b5-8664-0931881b390a","Type":"ContainerStarted","Data":"3684391e657aa22e0107a899d7b2741461bd9accb3494a8a275f18f9b929ba43"}
Nov 02 02:30:48.194374 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:48.194315    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-dn28l" event={"ID":"5aa7252e-e373-47b5-8664-0931881b390a","Type":"ContainerStarted","Data":"eedb17cf096e33087ef6e3548cbe07bb9a2de26990b1c87ea6bb75947ca681bd"}
Nov 02 02:30:48.326454 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:48.326006    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-dn28l" podStartSLOduration=2.325971642 podStartE2EDuration="2.325971642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:30:48.322836067 +0000 UTC m=+3913.226518182" watchObservedRunningTime="2026-11-02 02:30:48.325971642 +0000 UTC m=+3913.229653761"
Nov 02 02:30:49.120309 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:49.116423    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:30:49.120309 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:49.116520    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7" podUID="70ec0d83-9e6c-4e20-82b6-2f09909fc85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:30:49.247976 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:49.247395    5639 generic.go:356] "Generic (PLEG): container finished" podID="5aa7252e-e373-47b5-8664-0931881b390a" containerID="eedb17cf096e33087ef6e3548cbe07bb9a2de26990b1c87ea6bb75947ca681bd" exitCode=0
Nov 02 02:30:49.251227 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:49.251183    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-dn28l" event={"ID":"5aa7252e-e373-47b5-8664-0931881b390a","Type":"ContainerDied","Data":"eedb17cf096e33087ef6e3548cbe07bb9a2de26990b1c87ea6bb75947ca681bd"}
Nov 02 02:30:49.256463 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:49.256419    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:30:49.256608 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:49.256477    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx" podUID="db2e4537-de21-48f8-858e-1ead87636161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:30:50.386743 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.386714    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-dn28l"
Nov 02 02:30:50.404100 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.404043    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb" podUID="6bb83e70-1742-448d-abc9-a4dc65b6b970" containerName="main" containerID="cri-o://bc4fa6516ad6f38e30d6b7c7afc56f74135e7d0e95ea8591459e22be3f13aef7" gracePeriod=45
Nov 02 02:30:50.467339 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.467301    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb" podUID="6bb83e70-1742-448d-abc9-a4dc65b6b970" containerName="main" probeResult="failure" output="Get \"https://10.134.1.49:8000/ready\": dial tcp 10.134.1.49:8000: connect: connection refused"
Nov 02 02:30:50.498851 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.498820    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aa7252e-e373-47b5-8664-0931881b390a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa7252e-e373-47b5-8664-0931881b390a-maas-api-openshift-ca\") pod \"5aa7252e-e373-47b5-8664-0931881b390a\" (UID: \"5aa7252e-e373-47b5-8664-0931881b390a\") "
Nov 02 02:30:50.498999 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.498919    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5aa7252e-e373-47b5-8664-0931881b390a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa7252e-e373-47b5-8664-0931881b390a-maas-api-xks-ca\") pod \"5aa7252e-e373-47b5-8664-0931881b390a\" (UID: \"5aa7252e-e373-47b5-8664-0931881b390a\") "
Nov 02 02:30:50.499112 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.499078    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aa7252e-e373-47b5-8664-0931881b390a-kube-api-access-47x45\" (UniqueName: \"kubernetes.io/projected/5aa7252e-e373-47b5-8664-0931881b390a-kube-api-access-47x45\") pod \"5aa7252e-e373-47b5-8664-0931881b390a\" (UID: \"5aa7252e-e373-47b5-8664-0931881b390a\") "
Nov 02 02:30:50.499238 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.499217    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aa7252e-e373-47b5-8664-0931881b390a-maas-api-openshift-ca" pod "5aa7252e-e373-47b5-8664-0931881b390a" (UID: "5aa7252e-e373-47b5-8664-0931881b390a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:30:50.499457 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.499437    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa7252e-e373-47b5-8664-0931881b390a-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:30:50.501173 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.501148    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aa7252e-e373-47b5-8664-0931881b390a-maas-api-xks-ca" pod "5aa7252e-e373-47b5-8664-0931881b390a" (UID: "5aa7252e-e373-47b5-8664-0931881b390a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:30:50.501293 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.501261    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa7252e-e373-47b5-8664-0931881b390a-kube-api-access-47x45" pod "5aa7252e-e373-47b5-8664-0931881b390a" (UID: "5aa7252e-e373-47b5-8664-0931881b390a"). InnerVolumeSpecName "kube-api-access-47x45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:30:50.600182 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.600147    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47x45\" (UniqueName: \"kubernetes.io/projected/5aa7252e-e373-47b5-8664-0931881b390a-kube-api-access-47x45\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:30:50.600182 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.600179    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa7252e-e373-47b5-8664-0931881b390a-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:30:50.648991 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.648964    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"
Nov 02 02:30:50.701225 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.701189    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-dshm\") pod \"6bb83e70-1742-448d-abc9-a4dc65b6b970\" (UID: \"6bb83e70-1742-448d-abc9-a4dc65b6b970\") "
Nov 02 02:30:50.701404 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.701338    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-home\" (UniqueName: \"kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-home\") pod \"6bb83e70-1742-448d-abc9-a4dc65b6b970\" (UID: \"6bb83e70-1742-448d-abc9-a4dc65b6b970\") "
Nov 02 02:30:50.701404 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.701389    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-tmp-dir\") pod \"6bb83e70-1742-448d-abc9-a4dc65b6b970\" (UID: \"6bb83e70-1742-448d-abc9-a4dc65b6b970\") "
Nov 02 02:30:50.701491 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.701429    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bb83e70-1742-448d-abc9-a4dc65b6b970-tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb83e70-1742-448d-abc9-a4dc65b6b970-tls-certs\") pod \"6bb83e70-1742-448d-abc9-a4dc65b6b970\" (UID: \"6bb83e70-1742-448d-abc9-a4dc65b6b970\") "
Nov 02 02:30:50.701491 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.701467    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bb83e70-1742-448d-abc9-a4dc65b6b970-kube-api-access-646s2\" (UniqueName: \"kubernetes.io/projected/6bb83e70-1742-448d-abc9-a4dc65b6b970-kube-api-access-646s2\") pod \"6bb83e70-1742-448d-abc9-a4dc65b6b970\" (UID: \"6bb83e70-1742-448d-abc9-a4dc65b6b970\") "
Nov 02 02:30:50.701601 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.701517    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-model-cache\") pod \"6bb83e70-1742-448d-abc9-a4dc65b6b970\" (UID: \"6bb83e70-1742-448d-abc9-a4dc65b6b970\") "
Nov 02 02:30:50.701665 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.701620    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-home" pod "6bb83e70-1742-448d-abc9-a4dc65b6b970" (UID: "6bb83e70-1742-448d-abc9-a4dc65b6b970"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:30:50.701735 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.701673    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-tmp-dir" pod "6bb83e70-1742-448d-abc9-a4dc65b6b970" (UID: "6bb83e70-1742-448d-abc9-a4dc65b6b970"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:30:50.701925 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.701891    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-model-cache" pod "6bb83e70-1742-448d-abc9-a4dc65b6b970" (UID: "6bb83e70-1742-448d-abc9-a4dc65b6b970"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:30:50.702059 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.701963    5639 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-home\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:30:50.702059 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.702008    5639 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-tmp-dir\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:30:50.703740 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.703719    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-dshm" pod "6bb83e70-1742-448d-abc9-a4dc65b6b970" (UID: "6bb83e70-1742-448d-abc9-a4dc65b6b970"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:30:50.703827 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.703795    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb83e70-1742-448d-abc9-a4dc65b6b970-kube-api-access-646s2" pod "6bb83e70-1742-448d-abc9-a4dc65b6b970" (UID: "6bb83e70-1742-448d-abc9-a4dc65b6b970"). InnerVolumeSpecName "kube-api-access-646s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:30:50.703827 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.703806    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bb83e70-1742-448d-abc9-a4dc65b6b970-tls-certs" pod "6bb83e70-1742-448d-abc9-a4dc65b6b970" (UID: "6bb83e70-1742-448d-abc9-a4dc65b6b970"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:30:50.803355 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.803199    5639 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-dshm\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:30:50.803355 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.803226    5639 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb83e70-1742-448d-abc9-a4dc65b6b970-tls-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:30:50.803355 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.803239    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-646s2\" (UniqueName: \"kubernetes.io/projected/6bb83e70-1742-448d-abc9-a4dc65b6b970-kube-api-access-646s2\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:30:50.803355 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:50.803253    5639 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bb83e70-1742-448d-abc9-a4dc65b6b970-model-cache\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:30:51.258869 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:51.258835    5639 generic.go:356] "Generic (PLEG): container finished" podID="6bb83e70-1742-448d-abc9-a4dc65b6b970" containerID="bc4fa6516ad6f38e30d6b7c7afc56f74135e7d0e95ea8591459e22be3f13aef7" exitCode=0
Nov 02 02:30:51.259053 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:51.258882    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"
Nov 02 02:30:51.259053 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:51.258923    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb" event={"ID":"6bb83e70-1742-448d-abc9-a4dc65b6b970","Type":"ContainerDied","Data":"bc4fa6516ad6f38e30d6b7c7afc56f74135e7d0e95ea8591459e22be3f13aef7"}
Nov 02 02:30:51.259053 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:51.258963    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb" event={"ID":"6bb83e70-1742-448d-abc9-a4dc65b6b970","Type":"ContainerDied","Data":"f3ba905d14ac1c15142a5c255cab4709419992996541f1ff1f3ee7014c3bc111"}
Nov 02 02:30:51.259053 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:51.258985    5639 scope.go:122] "RemoveContainer" containerID="bc4fa6516ad6f38e30d6b7c7afc56f74135e7d0e95ea8591459e22be3f13aef7"
Nov 02 02:30:51.260497 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:51.260483    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-dn28l"
Nov 02 02:30:51.260583 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:51.260503    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-dn28l" event={"ID":"5aa7252e-e373-47b5-8664-0931881b390a","Type":"ContainerDied","Data":"3684391e657aa22e0107a899d7b2741461bd9accb3494a8a275f18f9b929ba43"}
Nov 02 02:30:51.260583 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:51.260550    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3684391e657aa22e0107a899d7b2741461bd9accb3494a8a275f18f9b929ba43"
Nov 02 02:30:51.268130 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:51.268111    5639 scope.go:122] "RemoveContainer" containerID="bc4fa6516ad6f38e30d6b7c7afc56f74135e7d0e95ea8591459e22be3f13aef7"
Nov 02 02:30:51.268447 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:30:51.268427    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc4fa6516ad6f38e30d6b7c7afc56f74135e7d0e95ea8591459e22be3f13aef7\": container with ID starting with bc4fa6516ad6f38e30d6b7c7afc56f74135e7d0e95ea8591459e22be3f13aef7 not found: ID does not exist" containerID="bc4fa6516ad6f38e30d6b7c7afc56f74135e7d0e95ea8591459e22be3f13aef7"
Nov 02 02:30:51.268521 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:51.268456    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc4fa6516ad6f38e30d6b7c7afc56f74135e7d0e95ea8591459e22be3f13aef7"} err="failed to get container status \"bc4fa6516ad6f38e30d6b7c7afc56f74135e7d0e95ea8591459e22be3f13aef7\": rpc error: code = NotFound desc = could not find container \"bc4fa6516ad6f38e30d6b7c7afc56f74135e7d0e95ea8591459e22be3f13aef7\": container with ID starting with bc4fa6516ad6f38e30d6b7c7afc56f74135e7d0e95ea8591459e22be3f13aef7 not found: ID does not exist"
Nov 02 02:30:51.422573 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:51.422536    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"]
Nov 02 02:30:51.429004 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:51.428978    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-eefc7478-kserve-648fbfdccc-4xwwb"]
Nov 02 02:30:52.791945 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:52.791906    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:30:53.217407 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:53.217371    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bb83e70-1742-448d-abc9-a4dc65b6b970" path="/var/lib/kubelet/pods/6bb83e70-1742-448d-abc9-a4dc65b6b970/volumes"
Nov 02 02:30:57.169594 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:57.169544    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m" podUID="6b43fa6c-393b-4f14-bc93-ee58fae93cce" containerName="main" containerID="cri-o://80a2dc24b786be1e587d8d69f9d1c7e2b865cb928c830f49138bcc3c56ff91fa" gracePeriod=45
Nov 02 02:30:57.300377 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:57.300343    5639 generic.go:356] "Generic (PLEG): container finished" podID="6b43fa6c-393b-4f14-bc93-ee58fae93cce" containerID="80a2dc24b786be1e587d8d69f9d1c7e2b865cb928c830f49138bcc3c56ff91fa" exitCode=0
Nov 02 02:30:57.300552 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:57.300416    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m" event={"ID":"6b43fa6c-393b-4f14-bc93-ee58fae93cce","Type":"ContainerDied","Data":"80a2dc24b786be1e587d8d69f9d1c7e2b865cb928c830f49138bcc3c56ff91fa"}
Nov 02 02:30:57.421893 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:30:57.421818    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:30:57.432813 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:57.432788    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"
Nov 02 02:30:57.476848 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:57.476816    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-tmp-dir\") pod \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\" (UID: \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\") "
Nov 02 02:30:57.477043 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:57.476931    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-dshm\") pod \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\" (UID: \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\") "
Nov 02 02:30:57.477043 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:57.476967    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-home\" (UniqueName: \"kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-home\") pod \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\" (UID: \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\") "
Nov 02 02:30:57.477043 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:57.476988    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b43fa6c-393b-4f14-bc93-ee58fae93cce-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b43fa6c-393b-4f14-bc93-ee58fae93cce-tls-certs\") pod \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\" (UID: \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\") "
Nov 02 02:30:57.477043 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:57.477016    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b43fa6c-393b-4f14-bc93-ee58fae93cce-kube-api-access-xpzd9\" (UniqueName: \"kubernetes.io/projected/6b43fa6c-393b-4f14-bc93-ee58fae93cce-kube-api-access-xpzd9\") pod \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\" (UID: \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\") "
Nov 02 02:30:57.477043 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:57.477033    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-model-cache\") pod \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\" (UID: \"6b43fa6c-393b-4f14-bc93-ee58fae93cce\") "
Nov 02 02:30:57.477347 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:57.477152    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-tmp-dir" pod "6b43fa6c-393b-4f14-bc93-ee58fae93cce" (UID: "6b43fa6c-393b-4f14-bc93-ee58fae93cce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:30:57.477410 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:57.477345    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-home" pod "6b43fa6c-393b-4f14-bc93-ee58fae93cce" (UID: "6b43fa6c-393b-4f14-bc93-ee58fae93cce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:30:57.477469 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:57.477443    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-model-cache" pod "6b43fa6c-393b-4f14-bc93-ee58fae93cce" (UID: "6b43fa6c-393b-4f14-bc93-ee58fae93cce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:30:57.477559 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:57.477545    5639 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-tmp-dir\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:30:57.477621 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:57.477560    5639 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-home\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:30:57.477621 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:57.477569    5639 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-model-cache\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:30:57.479364 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:57.479337    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b43fa6c-393b-4f14-bc93-ee58fae93cce-tls-certs" pod "6b43fa6c-393b-4f14-bc93-ee58fae93cce" (UID: "6b43fa6c-393b-4f14-bc93-ee58fae93cce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:30:57.479485 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:57.479366    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-dshm" pod "6b43fa6c-393b-4f14-bc93-ee58fae93cce" (UID: "6b43fa6c-393b-4f14-bc93-ee58fae93cce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:30:57.479485 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:57.479425    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b43fa6c-393b-4f14-bc93-ee58fae93cce-kube-api-access-xpzd9" pod "6b43fa6c-393b-4f14-bc93-ee58fae93cce" (UID: "6b43fa6c-393b-4f14-bc93-ee58fae93cce"). InnerVolumeSpecName "kube-api-access-xpzd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:30:57.578546 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:57.578509    5639 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b43fa6c-393b-4f14-bc93-ee58fae93cce-dshm\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:30:57.578546 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:57.578538    5639 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b43fa6c-393b-4f14-bc93-ee58fae93cce-tls-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:30:57.578546 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:57.578548    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpzd9\" (UniqueName: \"kubernetes.io/projected/6b43fa6c-393b-4f14-bc93-ee58fae93cce-kube-api-access-xpzd9\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:30:58.306477 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:58.306445    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"
Nov 02 02:30:58.306995 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:58.306484    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m" event={"ID":"6b43fa6c-393b-4f14-bc93-ee58fae93cce","Type":"ContainerDied","Data":"95e8dd85a910a9ae77858e3e0a1e1c468d13919bbbb771d9c5cc06df35146ac6"}
Nov 02 02:30:58.306995 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:58.306539    5639 scope.go:122] "RemoveContainer" containerID="80a2dc24b786be1e587d8d69f9d1c7e2b865cb928c830f49138bcc3c56ff91fa"
Nov 02 02:30:58.470860 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:58.470824    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"]
Nov 02 02:30:58.520490 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:58.520452    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-eefc7478-kserve-dfd768d57-qf65m"]
Nov 02 02:30:59.217803 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:30:59.217768    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b43fa6c-393b-4f14-bc93-ee58fae93cce" path="/var/lib/kubelet/pods/6b43fa6c-393b-4f14-bc93-ee58fae93cce/volumes"
Nov 02 02:31:04.496046 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:04.496011    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:31:12.968453 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:12.968413    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0320394b-gpzw4"]
Nov 02 02:31:12.968938 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:12.968923    5639 cpu_manager.go:414] "containerMap: removing container" podUID="6b43fa6c-393b-4f14-bc93-ee58fae93cce" containerName="main"
Nov 02 02:31:12.969031 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:12.968939    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b43fa6c-393b-4f14-bc93-ee58fae93cce" containerName="main"
Nov 02 02:31:12.969031 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:12.968951    5639 cpu_manager.go:414] "containerMap: removing container" podUID="5aa7252e-e373-47b5-8664-0931881b390a" containerName="revoke-keys"
Nov 02 02:31:12.969031 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:12.968958    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa7252e-e373-47b5-8664-0931881b390a" containerName="revoke-keys"
Nov 02 02:31:12.969031 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:12.968988    5639 cpu_manager.go:414] "containerMap: removing container" podUID="6bb83e70-1742-448d-abc9-a4dc65b6b970" containerName="main"
Nov 02 02:31:12.969031 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:12.968995    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb83e70-1742-448d-abc9-a4dc65b6b970" containerName="main"
Nov 02 02:31:12.969268 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:12.969080    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb83e70-1742-448d-abc9-a4dc65b6b970" containerName="main"
Nov 02 02:31:12.969268 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:12.969095    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b43fa6c-393b-4f14-bc93-ee58fae93cce" containerName="main"
Nov 02 02:31:12.969268 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:12.969101    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa7252e-e373-47b5-8664-0931881b390a" containerName="revoke-keys"
Nov 02 02:31:12.972352 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:12.972333    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0320394b-gpzw4"
Nov 02 02:31:13.025120 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:13.025085    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d03c8de-63cc-4af1-b612-05075329a1a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0320394b-gpzw4\" (UID: \"1d03c8de-63cc-4af1-b612-05075329a1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0320394b-gpzw4"
Nov 02 02:31:13.025120 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:13.025121    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d03c8de-63cc-4af1-b612-05075329a1a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0320394b-gpzw4\" (UID: \"1d03c8de-63cc-4af1-b612-05075329a1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0320394b-gpzw4"
Nov 02 02:31:13.025361 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:13.025211    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nfbt\" (UniqueName: \"kubernetes.io/projected/1d03c8de-63cc-4af1-b612-05075329a1a8-kube-api-access-7nfbt\") pod \"maas-api-revoke-sub-0320394b-gpzw4\" (UID: \"1d03c8de-63cc-4af1-b612-05075329a1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0320394b-gpzw4"
Nov 02 02:31:13.074183 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:13.074149    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:31:13.074183 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:13.074171    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:31:13.074183 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:13.074150    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:31:13.075995 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:13.075975    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0320394b-gpzw4"]
Nov 02 02:31:13.124577 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:13.124538    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:31:13.126082 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:13.126058    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d03c8de-63cc-4af1-b612-05075329a1a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0320394b-gpzw4\" (UID: \"1d03c8de-63cc-4af1-b612-05075329a1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0320394b-gpzw4"
Nov 02 02:31:13.126256 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:13.126089    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d03c8de-63cc-4af1-b612-05075329a1a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0320394b-gpzw4\" (UID: \"1d03c8de-63cc-4af1-b612-05075329a1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0320394b-gpzw4"
Nov 02 02:31:13.126256 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:13.126145    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nfbt\" (UniqueName: \"kubernetes.io/projected/1d03c8de-63cc-4af1-b612-05075329a1a8-kube-api-access-7nfbt\") pod \"maas-api-revoke-sub-0320394b-gpzw4\" (UID: \"1d03c8de-63cc-4af1-b612-05075329a1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0320394b-gpzw4"
Nov 02 02:31:13.126949 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:13.126911    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d03c8de-63cc-4af1-b612-05075329a1a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0320394b-gpzw4\" (UID: \"1d03c8de-63cc-4af1-b612-05075329a1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0320394b-gpzw4"
Nov 02 02:31:13.128839 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:13.128819    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d03c8de-63cc-4af1-b612-05075329a1a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0320394b-gpzw4\" (UID: \"1d03c8de-63cc-4af1-b612-05075329a1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0320394b-gpzw4"
Nov 02 02:31:13.187153 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:13.187124    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nfbt\" (UniqueName: \"kubernetes.io/projected/1d03c8de-63cc-4af1-b612-05075329a1a8-kube-api-access-7nfbt\") pod \"maas-api-revoke-sub-0320394b-gpzw4\" (UID: \"1d03c8de-63cc-4af1-b612-05075329a1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0320394b-gpzw4"
Nov 02 02:31:13.285346 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:13.285255    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0320394b-gpzw4"
Nov 02 02:31:13.657938 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:13.657868    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0320394b-gpzw4"]
Nov 02 02:31:13.660426 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:31:13.660397    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d03c8de_63cc_4af1_b612_05075329a1a8.slice/crio-69abfd5a8ebbdf9f693e95013a39f6a61b45108d7f5d5b29d1a1997bf094d75a WatchSource:0}: Error finding container 69abfd5a8ebbdf9f693e95013a39f6a61b45108d7f5d5b29d1a1997bf094d75a: Status 404 returned error can't find the container with id 69abfd5a8ebbdf9f693e95013a39f6a61b45108d7f5d5b29d1a1997bf094d75a
Nov 02 02:31:14.444589 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:14.444549    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0320394b-gpzw4" event={"ID":"1d03c8de-63cc-4af1-b612-05075329a1a8","Type":"ContainerStarted","Data":"ab75c70cbed149161aa7241e1cfb31af333c2ca757a2a3342e13bbec3e491f40"}
Nov 02 02:31:14.445009 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:14.444607    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0320394b-gpzw4" event={"ID":"1d03c8de-63cc-4af1-b612-05075329a1a8","Type":"ContainerStarted","Data":"69abfd5a8ebbdf9f693e95013a39f6a61b45108d7f5d5b29d1a1997bf094d75a"}
Nov 02 02:31:14.565406 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:14.565313    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0320394b-gpzw4" podStartSLOduration=2.565274539 podStartE2EDuration="2.565274539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:31:14.564592656 +0000 UTC m=+3939.468274769" watchObservedRunningTime="2026-11-02 02:31:14.565274539 +0000 UTC m=+3939.468956655"
Nov 02 02:31:15.450914 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:15.450880    5639 generic.go:356] "Generic (PLEG): container finished" podID="1d03c8de-63cc-4af1-b612-05075329a1a8" containerID="ab75c70cbed149161aa7241e1cfb31af333c2ca757a2a3342e13bbec3e491f40" exitCode=0
Nov 02 02:31:15.451385 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:15.450971    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0320394b-gpzw4" event={"ID":"1d03c8de-63cc-4af1-b612-05075329a1a8","Type":"ContainerDied","Data":"ab75c70cbed149161aa7241e1cfb31af333c2ca757a2a3342e13bbec3e491f40"}
Nov 02 02:31:16.595043 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:16.595015    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0320394b-gpzw4"
Nov 02 02:31:16.661671 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:16.661641    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d03c8de-63cc-4af1-b612-05075329a1a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d03c8de-63cc-4af1-b612-05075329a1a8-maas-api-xks-ca\") pod \"1d03c8de-63cc-4af1-b612-05075329a1a8\" (UID: \"1d03c8de-63cc-4af1-b612-05075329a1a8\") "
Nov 02 02:31:16.661873 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:16.661695    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d03c8de-63cc-4af1-b612-05075329a1a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d03c8de-63cc-4af1-b612-05075329a1a8-maas-api-openshift-ca\") pod \"1d03c8de-63cc-4af1-b612-05075329a1a8\" (UID: \"1d03c8de-63cc-4af1-b612-05075329a1a8\") "
Nov 02 02:31:16.661873 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:16.661772    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d03c8de-63cc-4af1-b612-05075329a1a8-kube-api-access-7nfbt\" (UniqueName: \"kubernetes.io/projected/1d03c8de-63cc-4af1-b612-05075329a1a8-kube-api-access-7nfbt\") pod \"1d03c8de-63cc-4af1-b612-05075329a1a8\" (UID: \"1d03c8de-63cc-4af1-b612-05075329a1a8\") "
Nov 02 02:31:16.662134 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:16.662107    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d03c8de-63cc-4af1-b612-05075329a1a8-maas-api-openshift-ca" pod "1d03c8de-63cc-4af1-b612-05075329a1a8" (UID: "1d03c8de-63cc-4af1-b612-05075329a1a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:31:16.664022 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:16.663996    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d03c8de-63cc-4af1-b612-05075329a1a8-maas-api-xks-ca" pod "1d03c8de-63cc-4af1-b612-05075329a1a8" (UID: "1d03c8de-63cc-4af1-b612-05075329a1a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:31:16.664138 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:16.664020    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d03c8de-63cc-4af1-b612-05075329a1a8-kube-api-access-7nfbt" pod "1d03c8de-63cc-4af1-b612-05075329a1a8" (UID: "1d03c8de-63cc-4af1-b612-05075329a1a8"). InnerVolumeSpecName "kube-api-access-7nfbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:31:16.763374 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:16.763264    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d03c8de-63cc-4af1-b612-05075329a1a8-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:31:16.763374 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:16.763319    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nfbt\" (UniqueName: \"kubernetes.io/projected/1d03c8de-63cc-4af1-b612-05075329a1a8-kube-api-access-7nfbt\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:31:16.763374 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:16.763331    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d03c8de-63cc-4af1-b612-05075329a1a8-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:31:17.462742 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:17.462708    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0320394b-gpzw4"
Nov 02 02:31:17.462742 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:17.462734    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0320394b-gpzw4" event={"ID":"1d03c8de-63cc-4af1-b612-05075329a1a8","Type":"ContainerDied","Data":"69abfd5a8ebbdf9f693e95013a39f6a61b45108d7f5d5b29d1a1997bf094d75a"}
Nov 02 02:31:17.463021 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:17.462785    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69abfd5a8ebbdf9f693e95013a39f6a61b45108d7f5d5b29d1a1997bf094d75a"
Nov 02 02:31:25.794889 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:25.794852    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:31:26.278156 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:26.278123    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p"]
Nov 02 02:31:26.278674 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:26.278655    5639 cpu_manager.go:414] "containerMap: removing container" podUID="1d03c8de-63cc-4af1-b612-05075329a1a8" containerName="revoke-keys"
Nov 02 02:31:26.278674 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:26.278677    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d03c8de-63cc-4af1-b612-05075329a1a8" containerName="revoke-keys"
Nov 02 02:31:26.278953 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:26.278816    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d03c8de-63cc-4af1-b612-05075329a1a8" containerName="revoke-keys"
Nov 02 02:31:26.282528 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:26.282504    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p"
Nov 02 02:31:26.384633 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:26.384589    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nk57n\""
Nov 02 02:31:26.384633 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:26.384589    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 02 02:31:26.384857 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:26.384658    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 02:31:26.386844 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:26.386820    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p"]
Nov 02 02:31:26.434540 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:26.434503    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:31:26.434540 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:26.434522    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:31:26.464977 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:26.464934    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj2gd\" (UniqueName: \"kubernetes.io/projected/a86bc454-99a9-47a5-bce7-85fb2277dffb-kube-api-access-vj2gd\") pod \"maas-api-f5655ff5-8sk9p\" (UID: \"a86bc454-99a9-47a5-bce7-85fb2277dffb\") " pod="odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p"
Nov 02 02:31:26.465302 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:26.465252    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a86bc454-99a9-47a5-bce7-85fb2277dffb-openshift-service-ca\") pod \"maas-api-f5655ff5-8sk9p\" (UID: \"a86bc454-99a9-47a5-bce7-85fb2277dffb\") " pod="odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p"
Nov 02 02:31:26.465529 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:26.465509    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a86bc454-99a9-47a5-bce7-85fb2277dffb-metrics-certs\") pod \"maas-api-f5655ff5-8sk9p\" (UID: \"a86bc454-99a9-47a5-bce7-85fb2277dffb\") " pod="odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p"
Nov 02 02:31:26.465747 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:26.465733    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a86bc454-99a9-47a5-bce7-85fb2277dffb-maas-api-tls\") pod \"maas-api-f5655ff5-8sk9p\" (UID: \"a86bc454-99a9-47a5-bce7-85fb2277dffb\") " pod="odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p"
Nov 02 02:31:26.567556 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:26.567464    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a86bc454-99a9-47a5-bce7-85fb2277dffb-metrics-certs\") pod \"maas-api-f5655ff5-8sk9p\" (UID: \"a86bc454-99a9-47a5-bce7-85fb2277dffb\") " pod="odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p"
Nov 02 02:31:26.567556 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:26.567542    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a86bc454-99a9-47a5-bce7-85fb2277dffb-maas-api-tls\") pod \"maas-api-f5655ff5-8sk9p\" (UID: \"a86bc454-99a9-47a5-bce7-85fb2277dffb\") " pod="odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p"
Nov 02 02:31:26.567813 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:26.567591    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj2gd\" (UniqueName: \"kubernetes.io/projected/a86bc454-99a9-47a5-bce7-85fb2277dffb-kube-api-access-vj2gd\") pod \"maas-api-f5655ff5-8sk9p\" (UID: \"a86bc454-99a9-47a5-bce7-85fb2277dffb\") " pod="odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p"
Nov 02 02:31:26.567813 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:26.567614    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a86bc454-99a9-47a5-bce7-85fb2277dffb-openshift-service-ca\") pod \"maas-api-f5655ff5-8sk9p\" (UID: \"a86bc454-99a9-47a5-bce7-85fb2277dffb\") " pod="odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p"
Nov 02 02:31:26.568242 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:26.568220    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a86bc454-99a9-47a5-bce7-85fb2277dffb-openshift-service-ca\") pod \"maas-api-f5655ff5-8sk9p\" (UID: \"a86bc454-99a9-47a5-bce7-85fb2277dffb\") " pod="odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p"
Nov 02 02:31:26.570179 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:26.570156    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a86bc454-99a9-47a5-bce7-85fb2277dffb-maas-api-tls\") pod \"maas-api-f5655ff5-8sk9p\" (UID: \"a86bc454-99a9-47a5-bce7-85fb2277dffb\") " pod="odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p"
Nov 02 02:31:26.570294 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:26.570197    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a86bc454-99a9-47a5-bce7-85fb2277dffb-metrics-certs\") pod \"maas-api-f5655ff5-8sk9p\" (UID: \"a86bc454-99a9-47a5-bce7-85fb2277dffb\") " pod="odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p"
Nov 02 02:31:26.628775 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:26.628745    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj2gd\" (UniqueName: \"kubernetes.io/projected/a86bc454-99a9-47a5-bce7-85fb2277dffb-kube-api-access-vj2gd\") pod \"maas-api-f5655ff5-8sk9p\" (UID: \"a86bc454-99a9-47a5-bce7-85fb2277dffb\") " pod="odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p"
Nov 02 02:31:26.895918 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:26.895885    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p"
Nov 02 02:31:27.274215 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:27.274189    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p"]
Nov 02 02:31:27.276320 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:31:27.276289    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda86bc454_99a9_47a5_bce7_85fb2277dffb.slice/crio-a729355cdcabf72752a7dd90d513da9109c50abc87d3e4c89bb8f00f9988e094 WatchSource:0}: Error finding container a729355cdcabf72752a7dd90d513da9109c50abc87d3e4c89bb8f00f9988e094: Status 404 returned error can't find the container with id a729355cdcabf72752a7dd90d513da9109c50abc87d3e4c89bb8f00f9988e094
Nov 02 02:31:27.277778 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:27.277759    5639 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 02:31:27.421052 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:31:27.421008    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:31:27.514519 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:27.514486    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p" event={"ID":"a86bc454-99a9-47a5-bce7-85fb2277dffb","Type":"ContainerStarted","Data":"a729355cdcabf72752a7dd90d513da9109c50abc87d3e4c89bb8f00f9988e094"}
Nov 02 02:31:29.528360 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:29.528247    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p" event={"ID":"a86bc454-99a9-47a5-bce7-85fb2277dffb","Type":"ContainerStarted","Data":"550684d3e7010e9ef9e967e5889f1279275d4a28e7017461dce1809d95c0c69a"}
Nov 02 02:31:29.528750 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:29.528373    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p"
Nov 02 02:31:29.646996 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:29.646936    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p" podStartSLOduration=1.768410891 podStartE2EDuration="3.646916667s" totalImagesPullingTime="1.878505776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:31:29.645578818 +0000 UTC m=+3954.549260932" watchObservedRunningTime="2026-11-02 02:31:29.646916667 +0000 UTC m=+3954.550598782"
Nov 02 02:31:33.647475 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.647439    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"]
Nov 02 02:31:33.651493 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.651471    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.704001 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.703858    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9ab719a4-openshift-default-dockercfg-wdp5p\""
Nov 02 02:31:33.740113 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.740081    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-envoy\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.740324 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.740143    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-token\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.740324 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.740243    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-workload-socket\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.740324 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.740272    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bbf788-40fb-43df-90b8-2d9074e07ce3-istiod-ca-cert\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.740446 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.740411    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-podinfo\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.740446 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.740438    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcdcz\" (UniqueName: \"kubernetes.io/projected/52bbf788-40fb-43df-90b8-2d9074e07ce3-kube-api-access-mcdcz\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.740514 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.740481    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-credential-socket\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.740550 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.740527    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-workload-certs\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.740589 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.740578    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-data\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.756762 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.756732    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"]
Nov 02 02:31:33.841895 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.841855    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-credential-socket\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.842113 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.841909    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-workload-certs\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.842113 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.841962    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-data\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.842113 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.841993    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-envoy\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.842113 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.842064    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-token\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.842473 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.842120    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-workload-socket\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.842473 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.842145    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bbf788-40fb-43df-90b8-2d9074e07ce3-istiod-ca-cert\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.842473 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.842180    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-podinfo\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.842473 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.842211    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcdcz\" (UniqueName: \"kubernetes.io/projected/52bbf788-40fb-43df-90b8-2d9074e07ce3-kube-api-access-mcdcz\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.842473 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.842274    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-credential-socket\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.842473 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.842359    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-data\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.842473 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.842408    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-workload-certs\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.842871 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.842684    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-workload-socket\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.843077 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.843053    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bbf788-40fb-43df-90b8-2d9074e07ce3-istiod-ca-cert\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.845070 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.845036    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-podinfo\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.845205 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.845047    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-envoy\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.903477 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.903444    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-token\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.903624 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.903593    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcdcz\" (UniqueName: \"kubernetes.io/projected/52bbf788-40fb-43df-90b8-2d9074e07ce3-kube-api-access-mcdcz\") pod \"e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") " pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:33.964449 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:33.964358    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:34.395412 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:34.395387    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"]
Nov 02 02:31:34.397675 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:31:34.397646    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bbf788_40fb_43df_90b8_2d9074e07ce3.slice/crio-8a6977856cf2e094ab3b03194ef5a32c8f394cb8ea23f124a687e5da6cb6e374 WatchSource:0}: Error finding container 8a6977856cf2e094ab3b03194ef5a32c8f394cb8ea23f124a687e5da6cb6e374: Status 404 returned error can't find the container with id 8a6977856cf2e094ab3b03194ef5a32c8f394cb8ea23f124a687e5da6cb6e374
Nov 02 02:31:34.400505 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:34.400469    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:31:34.400590 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:34.400538    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:31:34.400590 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:34.400567    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:31:34.556336 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:34.556272    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt" event={"ID":"52bbf788-40fb-43df-90b8-2d9074e07ce3","Type":"ContainerStarted","Data":"3580cab30d99b54ce4f19b4738c2820b102071f7ac22d1040523670e9009d5b3"}
Nov 02 02:31:34.556553 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:34.556373    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt" event={"ID":"52bbf788-40fb-43df-90b8-2d9074e07ce3","Type":"ContainerStarted","Data":"8a6977856cf2e094ab3b03194ef5a32c8f394cb8ea23f124a687e5da6cb6e374"}
Nov 02 02:31:34.677255 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:34.677060    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt" podStartSLOduration=1.6770409 podStartE2EDuration="1.6770409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:31:34.676892242 +0000 UTC m=+3959.580574369" watchObservedRunningTime="2026-11-02 02:31:34.6770409 +0000 UTC m=+3959.580723016"
Nov 02 02:31:34.965024 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:34.964935    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:35.110268 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:35.110217    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p"]
Nov 02 02:31:35.110591 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:35.110559    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p" podUID="a86bc454-99a9-47a5-bce7-85fb2277dffb" containerName="maas-api" containerID="cri-o://550684d3e7010e9ef9e967e5889f1279275d4a28e7017461dce1809d95c0c69a" gracePeriod=30
Nov 02 02:31:35.116472 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:35.116449    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p"
Nov 02 02:31:35.372187 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:35.372153    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p"
Nov 02 02:31:35.458815 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:35.458778    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a86bc454-99a9-47a5-bce7-85fb2277dffb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a86bc454-99a9-47a5-bce7-85fb2277dffb-openshift-service-ca\") pod \"a86bc454-99a9-47a5-bce7-85fb2277dffb\" (UID: \"a86bc454-99a9-47a5-bce7-85fb2277dffb\") "
Nov 02 02:31:35.459036 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:35.458848    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a86bc454-99a9-47a5-bce7-85fb2277dffb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a86bc454-99a9-47a5-bce7-85fb2277dffb-maas-api-tls\") pod \"a86bc454-99a9-47a5-bce7-85fb2277dffb\" (UID: \"a86bc454-99a9-47a5-bce7-85fb2277dffb\") "
Nov 02 02:31:35.459036 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:35.458916    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a86bc454-99a9-47a5-bce7-85fb2277dffb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a86bc454-99a9-47a5-bce7-85fb2277dffb-metrics-certs\") pod \"a86bc454-99a9-47a5-bce7-85fb2277dffb\" (UID: \"a86bc454-99a9-47a5-bce7-85fb2277dffb\") "
Nov 02 02:31:35.459036 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:35.458946    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a86bc454-99a9-47a5-bce7-85fb2277dffb-kube-api-access-vj2gd\" (UniqueName: \"kubernetes.io/projected/a86bc454-99a9-47a5-bce7-85fb2277dffb-kube-api-access-vj2gd\") pod \"a86bc454-99a9-47a5-bce7-85fb2277dffb\" (UID: \"a86bc454-99a9-47a5-bce7-85fb2277dffb\") "
Nov 02 02:31:35.459213 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:35.459118    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a86bc454-99a9-47a5-bce7-85fb2277dffb-openshift-service-ca" pod "a86bc454-99a9-47a5-bce7-85fb2277dffb" (UID: "a86bc454-99a9-47a5-bce7-85fb2277dffb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:31:35.459307 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:35.459248    5639 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a86bc454-99a9-47a5-bce7-85fb2277dffb-openshift-service-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:31:35.461266 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:35.461239    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a86bc454-99a9-47a5-bce7-85fb2277dffb-maas-api-tls" pod "a86bc454-99a9-47a5-bce7-85fb2277dffb" (UID: "a86bc454-99a9-47a5-bce7-85fb2277dffb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:31:35.461431 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:35.461295    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a86bc454-99a9-47a5-bce7-85fb2277dffb-metrics-certs" pod "a86bc454-99a9-47a5-bce7-85fb2277dffb" (UID: "a86bc454-99a9-47a5-bce7-85fb2277dffb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:31:35.461431 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:35.461328    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a86bc454-99a9-47a5-bce7-85fb2277dffb-kube-api-access-vj2gd" pod "a86bc454-99a9-47a5-bce7-85fb2277dffb" (UID: "a86bc454-99a9-47a5-bce7-85fb2277dffb"). InnerVolumeSpecName "kube-api-access-vj2gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:31:35.560586 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:35.560493    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj2gd\" (UniqueName: \"kubernetes.io/projected/a86bc454-99a9-47a5-bce7-85fb2277dffb-kube-api-access-vj2gd\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:31:35.560586 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:35.560530    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a86bc454-99a9-47a5-bce7-85fb2277dffb-maas-api-tls\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:31:35.560586 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:35.560547    5639 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a86bc454-99a9-47a5-bce7-85fb2277dffb-metrics-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:31:35.562944 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:35.562916    5639 generic.go:356] "Generic (PLEG): container finished" podID="a86bc454-99a9-47a5-bce7-85fb2277dffb" containerID="550684d3e7010e9ef9e967e5889f1279275d4a28e7017461dce1809d95c0c69a" exitCode=0
Nov 02 02:31:35.563091 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:35.562969    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p"
Nov 02 02:31:35.563091 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:35.562997    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p" event={"ID":"a86bc454-99a9-47a5-bce7-85fb2277dffb","Type":"ContainerDied","Data":"550684d3e7010e9ef9e967e5889f1279275d4a28e7017461dce1809d95c0c69a"}
Nov 02 02:31:35.563091 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:35.563058    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p" event={"ID":"a86bc454-99a9-47a5-bce7-85fb2277dffb","Type":"ContainerDied","Data":"a729355cdcabf72752a7dd90d513da9109c50abc87d3e4c89bb8f00f9988e094"}
Nov 02 02:31:35.563091 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:35.563087    5639 scope.go:122] "RemoveContainer" containerID="550684d3e7010e9ef9e967e5889f1279275d4a28e7017461dce1809d95c0c69a"
Nov 02 02:31:35.573408 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:35.573375    5639 scope.go:122] "RemoveContainer" containerID="550684d3e7010e9ef9e967e5889f1279275d4a28e7017461dce1809d95c0c69a"
Nov 02 02:31:35.573702 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:31:35.573674    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"550684d3e7010e9ef9e967e5889f1279275d4a28e7017461dce1809d95c0c69a\": container with ID starting with 550684d3e7010e9ef9e967e5889f1279275d4a28e7017461dce1809d95c0c69a not found: ID does not exist" containerID="550684d3e7010e9ef9e967e5889f1279275d4a28e7017461dce1809d95c0c69a"
Nov 02 02:31:35.573762 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:35.573713    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"550684d3e7010e9ef9e967e5889f1279275d4a28e7017461dce1809d95c0c69a"} err="failed to get container status \"550684d3e7010e9ef9e967e5889f1279275d4a28e7017461dce1809d95c0c69a\": rpc error: code = NotFound desc = could not find container \"550684d3e7010e9ef9e967e5889f1279275d4a28e7017461dce1809d95c0c69a\": container with ID starting with 550684d3e7010e9ef9e967e5889f1279275d4a28e7017461dce1809d95c0c69a not found: ID does not exist"
Nov 02 02:31:35.727257 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:35.727223    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p"]
Nov 02 02:31:35.733552 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:35.733523    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-f5655ff5-8sk9p"]
Nov 02 02:31:35.970634 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:35.970600    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:36.569855 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:36.569823    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:36.570812 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:36.570787    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:37.217592 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:37.217555    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a86bc454-99a9-47a5-bce7-85fb2277dffb" path="/var/lib/kubelet/pods/a86bc454-99a9-47a5-bce7-85fb2277dffb/volumes"
Nov 02 02:31:39.049073 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:39.049033    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d55g7"]
Nov 02 02:31:39.098939 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:39.098900    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d55g7"]
Nov 02 02:31:39.218043 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:39.218006    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e46eb581-da6b-46fe-beae-821c3b17bcd2" path="/var/lib/kubelet/pods/e46eb581-da6b-46fe-beae-821c3b17bcd2/volumes"
Nov 02 02:31:41.317969 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:41.317932    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m"]
Nov 02 02:31:41.318504 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:41.318484    5639 cpu_manager.go:414] "containerMap: removing container" podUID="a86bc454-99a9-47a5-bce7-85fb2277dffb" containerName="maas-api"
Nov 02 02:31:41.318588 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:41.318508    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a86bc454-99a9-47a5-bce7-85fb2277dffb" containerName="maas-api"
Nov 02 02:31:41.318655 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:41.318622    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="a86bc454-99a9-47a5-bce7-85fb2277dffb" containerName="maas-api"
Nov 02 02:31:41.321792 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:41.321774    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m"
Nov 02 02:31:41.374303 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:41.374240    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:31:41.423534 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:41.423465    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/457d8563-627c-4188-8927-9cb145661380-metrics-certs\") pod \"maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m\" (UID: \"457d8563-627c-4188-8927-9cb145661380\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m"
Nov 02 02:31:41.423720 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:41.423613    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/457d8563-627c-4188-8927-9cb145661380-maas-api-tls\") pod \"maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m\" (UID: \"457d8563-627c-4188-8927-9cb145661380\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m"
Nov 02 02:31:41.423720 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:41.423651    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/457d8563-627c-4188-8927-9cb145661380-openshift-service-ca\") pod \"maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m\" (UID: \"457d8563-627c-4188-8927-9cb145661380\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m"
Nov 02 02:31:41.423811 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:41.423760    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-682gm\" (UniqueName: \"kubernetes.io/projected/457d8563-627c-4188-8927-9cb145661380-kube-api-access-682gm\") pod \"maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m\" (UID: \"457d8563-627c-4188-8927-9cb145661380\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m"
Nov 02 02:31:41.423871 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:41.423826    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:31:41.423871 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:41.423842    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nk57n\""
Nov 02 02:31:41.423964 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:41.423915    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9ab719a4-serving-cert\""
Nov 02 02:31:41.424091 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:41.424077    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 02:31:41.425939 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:41.425918    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m"]
Nov 02 02:31:41.524756 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:41.524723    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/457d8563-627c-4188-8927-9cb145661380-maas-api-tls\") pod \"maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m\" (UID: \"457d8563-627c-4188-8927-9cb145661380\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m"
Nov 02 02:31:41.524977 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:41.524761    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/457d8563-627c-4188-8927-9cb145661380-openshift-service-ca\") pod \"maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m\" (UID: \"457d8563-627c-4188-8927-9cb145661380\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m"
Nov 02 02:31:41.524977 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:41.524803    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-682gm\" (UniqueName: \"kubernetes.io/projected/457d8563-627c-4188-8927-9cb145661380-kube-api-access-682gm\") pod \"maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m\" (UID: \"457d8563-627c-4188-8927-9cb145661380\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m"
Nov 02 02:31:41.524977 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:41.524867    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/457d8563-627c-4188-8927-9cb145661380-metrics-certs\") pod \"maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m\" (UID: \"457d8563-627c-4188-8927-9cb145661380\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m"
Nov 02 02:31:41.524977 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:31:41.524880    5639 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-serving-cert: secret "maas-api-e2e-ait-9ab719a4-serving-cert" not found
Nov 02 02:31:41.524977 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:31:41.524972    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/457d8563-627c-4188-8927-9cb145661380-maas-api-tls podName:457d8563-627c-4188-8927-9cb145661380 nodeName:}" failed. No retries permitted until 2026-11-02 02:31:42.024949139 +0000 UTC m=+3966.928631252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/457d8563-627c-4188-8927-9cb145661380-maas-api-tls") pod "maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m" (UID: "457d8563-627c-4188-8927-9cb145661380") : secret "maas-api-e2e-ait-9ab719a4-serving-cert" not found
Nov 02 02:31:41.525581 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:41.525558    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/457d8563-627c-4188-8927-9cb145661380-openshift-service-ca\") pod \"maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m\" (UID: \"457d8563-627c-4188-8927-9cb145661380\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m"
Nov 02 02:31:41.527458 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:41.527439    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/457d8563-627c-4188-8927-9cb145661380-metrics-certs\") pod \"maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m\" (UID: \"457d8563-627c-4188-8927-9cb145661380\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m"
Nov 02 02:31:41.588186 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:41.588148    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-682gm\" (UniqueName: \"kubernetes.io/projected/457d8563-627c-4188-8927-9cb145661380-kube-api-access-682gm\") pod \"maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m\" (UID: \"457d8563-627c-4188-8927-9cb145661380\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m"
Nov 02 02:31:42.031511 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:42.031421    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/457d8563-627c-4188-8927-9cb145661380-maas-api-tls\") pod \"maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m\" (UID: \"457d8563-627c-4188-8927-9cb145661380\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m"
Nov 02 02:31:42.034070 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:42.034043    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/457d8563-627c-4188-8927-9cb145661380-maas-api-tls\") pod \"maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m\" (UID: \"457d8563-627c-4188-8927-9cb145661380\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m"
Nov 02 02:31:42.234590 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:42.234556    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m"
Nov 02 02:31:42.608957 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:42.608931    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m"]
Nov 02 02:31:42.611671 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:31:42.611640    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457d8563_627c_4188_8927_9cb145661380.slice/crio-af51cb02dc5c04afa72da717269acfb743148fd7955cb24f266f0b361bbbf8d9 WatchSource:0}: Error finding container af51cb02dc5c04afa72da717269acfb743148fd7955cb24f266f0b361bbbf8d9: Status 404 returned error can't find the container with id af51cb02dc5c04afa72da717269acfb743148fd7955cb24f266f0b361bbbf8d9
Nov 02 02:31:43.608579 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:43.608544    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m" event={"ID":"457d8563-627c-4188-8927-9cb145661380","Type":"ContainerStarted","Data":"f4f5c74ed3422a1d6eec1b17293c3bdf880bcb2dab982466e15cee595ade1f96"}
Nov 02 02:31:43.608771 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:43.608599    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m" event={"ID":"457d8563-627c-4188-8927-9cb145661380","Type":"ContainerStarted","Data":"af51cb02dc5c04afa72da717269acfb743148fd7955cb24f266f0b361bbbf8d9"}
Nov 02 02:31:43.608771 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:43.608619    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m"
Nov 02 02:31:43.781597 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:43.781419    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m" podStartSLOduration=1.922223903 podStartE2EDuration="2.781401977s" totalImagesPullingTime="859.178074ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:31:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:31:43.781047723 +0000 UTC m=+3968.684729845" watchObservedRunningTime="2026-11-02 02:31:43.781401977 +0000 UTC m=+3968.685084086"
Nov 02 02:31:47.806772 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:47.806692    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m"]
Nov 02 02:31:47.810640 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:47.810621    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m"
Nov 02 02:31:47.868459 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:47.868422    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m"]
Nov 02 02:31:47.901810 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:47.901774    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b683b83-cd90-4840-87d6-158c702b9c50-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m\" (UID: \"3b683b83-cd90-4840-87d6-158c702b9c50\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m"
Nov 02 02:31:47.902036 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:47.901940    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klzj5\" (UniqueName: \"kubernetes.io/projected/3b683b83-cd90-4840-87d6-158c702b9c50-kube-api-access-klzj5\") pod \"payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m\" (UID: \"3b683b83-cd90-4840-87d6-158c702b9c50\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m"
Nov 02 02:31:47.915018 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:47.914996    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9ab719a4\""
Nov 02 02:31:47.970303 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:47.960372    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp"]
Nov 02 02:31:47.970303 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:47.966520    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp"
Nov 02 02:31:48.003205 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:48.003168    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klzj5\" (UniqueName: \"kubernetes.io/projected/3b683b83-cd90-4840-87d6-158c702b9c50-kube-api-access-klzj5\") pod \"payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m\" (UID: \"3b683b83-cd90-4840-87d6-158c702b9c50\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m"
Nov 02 02:31:48.003412 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:48.003235    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b683b83-cd90-4840-87d6-158c702b9c50-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m\" (UID: \"3b683b83-cd90-4840-87d6-158c702b9c50\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m"
Nov 02 02:31:48.003942 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:48.003923    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b683b83-cd90-4840-87d6-158c702b9c50-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m\" (UID: \"3b683b83-cd90-4840-87d6-158c702b9c50\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m"
Nov 02 02:31:48.019195 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:48.019159    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9ab719a4-dockercfg-sc6pn\""
Nov 02 02:31:48.021236 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:48.021208    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp"]
Nov 02 02:31:48.064598 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:48.064517    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klzj5\" (UniqueName: \"kubernetes.io/projected/3b683b83-cd90-4840-87d6-158c702b9c50-kube-api-access-klzj5\") pod \"payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m\" (UID: \"3b683b83-cd90-4840-87d6-158c702b9c50\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m"
Nov 02 02:31:48.104706 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:48.104673    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb863a90-ffbe-46fe-b167-88efe18365ea-plugins-config-volume\") pod \"payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp\" (UID: \"bb863a90-ffbe-46fe-b167-88efe18365ea\") " pod="openshift-ingress/payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp"
Nov 02 02:31:48.104877 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:48.104863    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct5qh\" (UniqueName: \"kubernetes.io/projected/bb863a90-ffbe-46fe-b167-88efe18365ea-kube-api-access-ct5qh\") pod \"payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp\" (UID: \"bb863a90-ffbe-46fe-b167-88efe18365ea\") " pod="openshift-ingress/payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp"
Nov 02 02:31:48.124476 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:48.124443    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m"
Nov 02 02:31:48.205488 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:48.205456    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct5qh\" (UniqueName: \"kubernetes.io/projected/bb863a90-ffbe-46fe-b167-88efe18365ea-kube-api-access-ct5qh\") pod \"payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp\" (UID: \"bb863a90-ffbe-46fe-b167-88efe18365ea\") " pod="openshift-ingress/payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp"
Nov 02 02:31:48.205677 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:48.205517    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb863a90-ffbe-46fe-b167-88efe18365ea-plugins-config-volume\") pod \"payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp\" (UID: \"bb863a90-ffbe-46fe-b167-88efe18365ea\") " pod="openshift-ingress/payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp"
Nov 02 02:31:48.206221 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:48.206199    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb863a90-ffbe-46fe-b167-88efe18365ea-plugins-config-volume\") pod \"payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp\" (UID: \"bb863a90-ffbe-46fe-b167-88efe18365ea\") " pod="openshift-ingress/payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp"
Nov 02 02:31:48.267946 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:48.267911    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct5qh\" (UniqueName: \"kubernetes.io/projected/bb863a90-ffbe-46fe-b167-88efe18365ea-kube-api-access-ct5qh\") pod \"payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp\" (UID: \"bb863a90-ffbe-46fe-b167-88efe18365ea\") " pod="openshift-ingress/payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp"
Nov 02 02:31:48.283829 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:48.283800    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp"
Nov 02 02:31:48.498696 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:48.498665    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m"]
Nov 02 02:31:48.500861 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:31:48.500832    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b683b83_cd90_4840_87d6_158c702b9c50.slice/crio-50f47124bd70eff21e62c6b2c17f713c186d181d887f2a272720c025f59e1f98 WatchSource:0}: Error finding container 50f47124bd70eff21e62c6b2c17f713c186d181d887f2a272720c025f59e1f98: Status 404 returned error can't find the container with id 50f47124bd70eff21e62c6b2c17f713c186d181d887f2a272720c025f59e1f98
Nov 02 02:31:48.635812 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:48.635724    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m" event={"ID":"3b683b83-cd90-4840-87d6-158c702b9c50","Type":"ContainerStarted","Data":"4235135d7e5eae97645dc37cc55ad29ac1ad62bd8c4cc8dedcfda4cf38886d70"}
Nov 02 02:31:48.635812 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:48.635778    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m" event={"ID":"3b683b83-cd90-4840-87d6-158c702b9c50","Type":"ContainerStarted","Data":"50f47124bd70eff21e62c6b2c17f713c186d181d887f2a272720c025f59e1f98"}
Nov 02 02:31:48.636021 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:48.635833    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m"
Nov 02 02:31:48.659826 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:48.659765    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp"]
Nov 02 02:31:48.662044 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:31:48.662021    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb863a90_ffbe_46fe_b167_88efe18365ea.slice/crio-e9693ccdb6f2d362d56d865efe5e9ccbb77bf4e0c294773190d0d3b5f5871323 WatchSource:0}: Error finding container e9693ccdb6f2d362d56d865efe5e9ccbb77bf4e0c294773190d0d3b5f5871323: Status 404 returned error can't find the container with id e9693ccdb6f2d362d56d865efe5e9ccbb77bf4e0c294773190d0d3b5f5871323
Nov 02 02:31:48.752732 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:48.752679    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m" podStartSLOduration=1.7526635069999998 podStartE2EDuration="1.752663507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:31:48.751058234 +0000 UTC m=+3973.654740349" watchObservedRunningTime="2026-11-02 02:31:48.752663507 +0000 UTC m=+3973.656345677"
Nov 02 02:31:49.615545 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:49.615504    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:31:49.619847 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:49.619820    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m"
Nov 02 02:31:49.643044 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:49.643006    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp" event={"ID":"bb863a90-ffbe-46fe-b167-88efe18365ea","Type":"ContainerStarted","Data":"99a88161a850fe6c0cfe087a3812219c756cecbf3a4b185382cda76977790f12"}
Nov 02 02:31:49.643237 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:49.643069    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp"
Nov 02 02:31:49.643237 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:49.643094    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp" event={"ID":"bb863a90-ffbe-46fe-b167-88efe18365ea","Type":"ContainerStarted","Data":"e9693ccdb6f2d362d56d865efe5e9ccbb77bf4e0c294773190d0d3b5f5871323"}
Nov 02 02:31:57.420876 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:31:57.420831    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:31:58.716601 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:58.716569    5639 generic.go:356] "Generic (PLEG): container finished" podID="52bbf788-40fb-43df-90b8-2d9074e07ce3" containerID="3580cab30d99b54ce4f19b4738c2820b102071f7ac22d1040523670e9009d5b3" exitCode=0
Nov 02 02:31:58.717065 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:58.716607    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt" event={"ID":"52bbf788-40fb-43df-90b8-2d9074e07ce3","Type":"ContainerDied","Data":"3580cab30d99b54ce4f19b4738c2820b102071f7ac22d1040523670e9009d5b3"}
Nov 02 02:31:58.717116 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:58.717063    5639 scope.go:122] "RemoveContainer" containerID="3580cab30d99b54ce4f19b4738c2820b102071f7ac22d1040523670e9009d5b3"
Nov 02 02:31:58.718503 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:58.718473    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:31:58.718597 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:58.718544    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:31:58.718597 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:58.718575    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:31:58.927164 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:58.927063    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp" podStartSLOduration=11.927048809 podStartE2EDuration="11.927048809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:31:49.856390524 +0000 UTC m=+3974.760072638" watchObservedRunningTime="2026-11-02 02:31:58.927048809 +0000 UTC m=+3983.830730972"
Nov 02 02:31:59.646431 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:59.646400    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m"
Nov 02 02:31:59.724631 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:59.724594    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt" event={"ID":"52bbf788-40fb-43df-90b8-2d9074e07ce3","Type":"ContainerStarted","Data":"d3a989734adc398615a7d1f4eaee3410f5d58b0d4a48131bb09eb438a480d5e8"}
Nov 02 02:31:59.968732 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:59.968634    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:31:59.969099 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:59.969074    5639 patch_prober.go:29] interesting pod/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.56:15021/healthz/ready\": dial tcp 10.134.1.56:15021: connect: connection refused" start-of-body=
Nov 02 02:31:59.969335 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:31:59.969308    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt" podUID="52bbf788-40fb-43df-90b8-2d9074e07ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.56:15021/healthz/ready\": dial tcp 10.134.1.56:15021: connect: connection refused"
Nov 02 02:32:00.652864 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:00.652834    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp"
Nov 02 02:32:00.979374 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:00.968743    5639 patch_prober.go:29] interesting pod/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.56:15021/healthz/ready\": dial tcp 10.134.1.56:15021: connect: connection refused" start-of-body=
Nov 02 02:32:00.979374 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:00.968811    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt" podUID="52bbf788-40fb-43df-90b8-2d9074e07ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.56:15021/healthz/ready\": dial tcp 10.134.1.56:15021: connect: connection refused"
Nov 02 02:32:01.969790 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:01.969747    5639 patch_prober.go:29] interesting pod/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.56:15021/healthz/ready\": dial tcp 10.134.1.56:15021: connect: connection refused" start-of-body=
Nov 02 02:32:01.970133 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:01.970093    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt" podUID="52bbf788-40fb-43df-90b8-2d9074e07ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.56:15021/healthz/ready\": dial tcp 10.134.1.56:15021: connect: connection refused"
Nov 02 02:32:02.971419 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:02.970428    5639 patch_prober.go:29] interesting pod/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.56:15021/healthz/ready\": dial tcp 10.134.1.56:15021: connect: connection refused" start-of-body=
Nov 02 02:32:02.971419 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:02.970490    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt" podUID="52bbf788-40fb-43df-90b8-2d9074e07ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.56:15021/healthz/ready\": dial tcp 10.134.1.56:15021: connect: connection refused"
Nov 02 02:32:03.971617 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:03.965712    5639 patch_prober.go:29] interesting pod/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.56:15021/healthz/ready\": dial tcp 10.134.1.56:15021: connect: connection refused" start-of-body=
Nov 02 02:32:03.971617 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:03.965778    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt" podUID="52bbf788-40fb-43df-90b8-2d9074e07ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.56:15021/healthz/ready\": dial tcp 10.134.1.56:15021: connect: connection refused"
Nov 02 02:32:04.969695 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:04.969664    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:32:04.970007 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:04.969986    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:32:04.970855 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:04.970833    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:32:08.046741 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:08.046704    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bbmw9"]
Nov 02 02:32:08.096011 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:08.095977    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bbmw9"]
Nov 02 02:32:09.218032 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:09.218002    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df0de6c6-be7b-4863-8f63-b6ca94e7a6e9" path="/var/lib/kubelet/pods/df0de6c6-be7b-4863-8f63-b6ca94e7a6e9/volumes"
Nov 02 02:32:13.396419 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:13.396381    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg"]
Nov 02 02:32:13.400083 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:13.400061    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg"
Nov 02 02:32:13.452354 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:13.452316    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:32:13.501898 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:13.501865    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhr9x\""
Nov 02 02:32:13.503872 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:13.503851    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg"]
Nov 02 02:32:13.526388 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:13.526350    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg\" (UID: \"a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg"
Nov 02 02:32:13.526552 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:13.526427    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp9m4\" (UniqueName: \"kubernetes.io/projected/a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e-kube-api-access-hp9m4\") pod \"maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg\" (UID: \"a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg"
Nov 02 02:32:13.526552 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:13.526525    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg\" (UID: \"a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg"
Nov 02 02:32:13.627289 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:13.627240    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp9m4\" (UniqueName: \"kubernetes.io/projected/a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e-kube-api-access-hp9m4\") pod \"maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg\" (UID: \"a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg"
Nov 02 02:32:13.627483 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:13.627363    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg\" (UID: \"a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg"
Nov 02 02:32:13.627527 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:13.627483    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg\" (UID: \"a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg"
Nov 02 02:32:13.628140 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:13.628119    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg\" (UID: \"a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg"
Nov 02 02:32:13.630037 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:13.630007    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg\" (UID: \"a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg"
Nov 02 02:32:13.690332 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:13.690238    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp9m4\" (UniqueName: \"kubernetes.io/projected/a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e-kube-api-access-hp9m4\") pod \"maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg\" (UID: \"a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg"
Nov 02 02:32:13.713111 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:13.713082    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg"
Nov 02 02:32:14.088023 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:14.087995    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg"]
Nov 02 02:32:14.090033 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:32:14.089999    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6fbbbfd_6df9_47fa_b6d9_03f5929e9e6e.slice/crio-395485077af83b7be7db8ae56f521cda5b4b57af5370ccbde12bc7969c2e2203 WatchSource:0}: Error finding container 395485077af83b7be7db8ae56f521cda5b4b57af5370ccbde12bc7969c2e2203: Status 404 returned error can't find the container with id 395485077af83b7be7db8ae56f521cda5b4b57af5370ccbde12bc7969c2e2203
Nov 02 02:32:14.805880 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:14.805792    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg" event={"ID":"a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e","Type":"ContainerStarted","Data":"949bf67d7dbdf98ee5e1125d150772990393483384f42a33f569874ddecde3f6"}
Nov 02 02:32:14.805880 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:14.805860    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg" event={"ID":"a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e","Type":"ContainerStarted","Data":"395485077af83b7be7db8ae56f521cda5b4b57af5370ccbde12bc7969c2e2203"}
Nov 02 02:32:14.922692 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:14.922639    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg" podStartSLOduration=1.922624565 podStartE2EDuration="1.922624565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:32:14.92081727 +0000 UTC m=+3999.824499385" watchObservedRunningTime="2026-11-02 02:32:14.922624565 +0000 UTC m=+3999.826306679"
Nov 02 02:32:15.813160 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:15.813121    5639 generic.go:356] "Generic (PLEG): container finished" podID="a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e" containerID="949bf67d7dbdf98ee5e1125d150772990393483384f42a33f569874ddecde3f6" exitCode=0
Nov 02 02:32:15.813675 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:15.813208    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg" event={"ID":"a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e","Type":"ContainerDied","Data":"949bf67d7dbdf98ee5e1125d150772990393483384f42a33f569874ddecde3f6"}
Nov 02 02:32:16.981526 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:16.981497    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg"
Nov 02 02:32:17.074647 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:17.074536    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e-maas-api-xks-ca\") pod \"a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e\" (UID: \"a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e\") "
Nov 02 02:32:17.074647 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:17.074639    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e-kube-api-access-hp9m4\" (UniqueName: \"kubernetes.io/projected/a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e-kube-api-access-hp9m4\") pod \"a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e\" (UID: \"a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e\") "
Nov 02 02:32:17.074910 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:17.074765    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e-maas-api-openshift-ca\") pod \"a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e\" (UID: \"a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e\") "
Nov 02 02:32:17.075167 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:17.075139    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e-maas-api-openshift-ca" pod "a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e" (UID: "a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:32:17.076896 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:17.076858    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e-maas-api-xks-ca" pod "a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e" (UID: "a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:32:17.076896 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:17.076877    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e-kube-api-access-hp9m4" pod "a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e" (UID: "a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e"). InnerVolumeSpecName "kube-api-access-hp9m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:32:17.176606 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:17.176557    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:17.176606 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:17.176600    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:17.176606 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:17.176611    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp9m4\" (UniqueName: \"kubernetes.io/projected/a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e-kube-api-access-hp9m4\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:17.825515 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:17.825481    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg" event={"ID":"a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e","Type":"ContainerDied","Data":"395485077af83b7be7db8ae56f521cda5b4b57af5370ccbde12bc7969c2e2203"}
Nov 02 02:32:17.825719 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:17.825535    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="395485077af83b7be7db8ae56f521cda5b4b57af5370ccbde12bc7969c2e2203"
Nov 02 02:32:17.825719 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:17.825490    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg"
Nov 02 02:32:24.869551 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:24.869515    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp"]
Nov 02 02:32:24.974006 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:24.973972    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m"]
Nov 02 02:32:27.420729 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:32:27.420684    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:32:29.870624 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:29.870559    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp" podUID="bb863a90-ffbe-46fe-b167-88efe18365ea" containerName="payload-processing" containerID="cri-o://99a88161a850fe6c0cfe087a3812219c756cecbf3a4b185382cda76977790f12" gracePeriod=25
Nov 02 02:32:29.974777 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:29.974714    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m" podUID="3b683b83-cd90-4840-87d6-158c702b9c50" containerName="payload-pre-processing" containerID="cri-o://4235135d7e5eae97645dc37cc55ad29ac1ad62bd8c4cc8dedcfda4cf38886d70" gracePeriod=25
Nov 02 02:32:30.121753 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.121681    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp"
Nov 02 02:32:30.213786 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.213756    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb863a90-ffbe-46fe-b167-88efe18365ea-kube-api-access-ct5qh\" (UniqueName: \"kubernetes.io/projected/bb863a90-ffbe-46fe-b167-88efe18365ea-kube-api-access-ct5qh\") pod \"bb863a90-ffbe-46fe-b167-88efe18365ea\" (UID: \"bb863a90-ffbe-46fe-b167-88efe18365ea\") "
Nov 02 02:32:30.213975 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.213891    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb863a90-ffbe-46fe-b167-88efe18365ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb863a90-ffbe-46fe-b167-88efe18365ea-plugins-config-volume\") pod \"bb863a90-ffbe-46fe-b167-88efe18365ea\" (UID: \"bb863a90-ffbe-46fe-b167-88efe18365ea\") "
Nov 02 02:32:30.214357 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.214326    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb863a90-ffbe-46fe-b167-88efe18365ea-plugins-config-volume" pod "bb863a90-ffbe-46fe-b167-88efe18365ea" (UID: "bb863a90-ffbe-46fe-b167-88efe18365ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:32:30.216357 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.216314    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb863a90-ffbe-46fe-b167-88efe18365ea-kube-api-access-ct5qh" pod "bb863a90-ffbe-46fe-b167-88efe18365ea" (UID: "bb863a90-ffbe-46fe-b167-88efe18365ea"). InnerVolumeSpecName "kube-api-access-ct5qh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:32:30.232931 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.232907    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m"
Nov 02 02:32:30.288778 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.288738    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m"]
Nov 02 02:32:30.288999 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.288976    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m" podUID="457d8563-627c-4188-8927-9cb145661380" containerName="maas-api" containerID="cri-o://f4f5c74ed3422a1d6eec1b17293c3bdf880bcb2dab982466e15cee595ade1f96" gracePeriod=30
Nov 02 02:32:30.315715 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.315677    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b683b83-cd90-4840-87d6-158c702b9c50-kube-api-access-klzj5\" (UniqueName: \"kubernetes.io/projected/3b683b83-cd90-4840-87d6-158c702b9c50-kube-api-access-klzj5\") pod \"3b683b83-cd90-4840-87d6-158c702b9c50\" (UID: \"3b683b83-cd90-4840-87d6-158c702b9c50\") "
Nov 02 02:32:30.315876 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.315852    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b683b83-cd90-4840-87d6-158c702b9c50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b683b83-cd90-4840-87d6-158c702b9c50-plugins-config-volume\") pod \"3b683b83-cd90-4840-87d6-158c702b9c50\" (UID: \"3b683b83-cd90-4840-87d6-158c702b9c50\") "
Nov 02 02:32:30.316375 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.316318    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb863a90-ffbe-46fe-b167-88efe18365ea-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:30.316375 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.316347    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct5qh\" (UniqueName: \"kubernetes.io/projected/bb863a90-ffbe-46fe-b167-88efe18365ea-kube-api-access-ct5qh\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:30.316607 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.316391    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b683b83-cd90-4840-87d6-158c702b9c50-plugins-config-volume" pod "3b683b83-cd90-4840-87d6-158c702b9c50" (UID: "3b683b83-cd90-4840-87d6-158c702b9c50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:32:30.318329 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.318295    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b683b83-cd90-4840-87d6-158c702b9c50-kube-api-access-klzj5" pod "3b683b83-cd90-4840-87d6-158c702b9c50" (UID: "3b683b83-cd90-4840-87d6-158c702b9c50"). InnerVolumeSpecName "kube-api-access-klzj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:32:30.417892 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.417857    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klzj5\" (UniqueName: \"kubernetes.io/projected/3b683b83-cd90-4840-87d6-158c702b9c50-kube-api-access-klzj5\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:30.418029 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.417896    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b683b83-cd90-4840-87d6-158c702b9c50-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:30.535621 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.535593    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m"
Nov 02 02:32:30.620728 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.620699    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/457d8563-627c-4188-8927-9cb145661380-metrics-certs\" (UniqueName: \"kubernetes.io/secret/457d8563-627c-4188-8927-9cb145661380-metrics-certs\") pod \"457d8563-627c-4188-8927-9cb145661380\" (UID: \"457d8563-627c-4188-8927-9cb145661380\") "
Nov 02 02:32:30.620728 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.620730    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/457d8563-627c-4188-8927-9cb145661380-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/457d8563-627c-4188-8927-9cb145661380-maas-api-tls\") pod \"457d8563-627c-4188-8927-9cb145661380\" (UID: \"457d8563-627c-4188-8927-9cb145661380\") "
Nov 02 02:32:30.621009 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.620800    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/457d8563-627c-4188-8927-9cb145661380-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/457d8563-627c-4188-8927-9cb145661380-openshift-service-ca\") pod \"457d8563-627c-4188-8927-9cb145661380\" (UID: \"457d8563-627c-4188-8927-9cb145661380\") "
Nov 02 02:32:30.621009 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.620823    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/457d8563-627c-4188-8927-9cb145661380-kube-api-access-682gm\" (UniqueName: \"kubernetes.io/projected/457d8563-627c-4188-8927-9cb145661380-kube-api-access-682gm\") pod \"457d8563-627c-4188-8927-9cb145661380\" (UID: \"457d8563-627c-4188-8927-9cb145661380\") "
Nov 02 02:32:30.621210 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.621182    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/457d8563-627c-4188-8927-9cb145661380-openshift-service-ca" pod "457d8563-627c-4188-8927-9cb145661380" (UID: "457d8563-627c-4188-8927-9cb145661380"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:32:30.623027 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.623004    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/457d8563-627c-4188-8927-9cb145661380-maas-api-tls" pod "457d8563-627c-4188-8927-9cb145661380" (UID: "457d8563-627c-4188-8927-9cb145661380"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:32:30.623126 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.623066    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457d8563-627c-4188-8927-9cb145661380-kube-api-access-682gm" pod "457d8563-627c-4188-8927-9cb145661380" (UID: "457d8563-627c-4188-8927-9cb145661380"). InnerVolumeSpecName "kube-api-access-682gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:32:30.623126 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.623078    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/457d8563-627c-4188-8927-9cb145661380-metrics-certs" pod "457d8563-627c-4188-8927-9cb145661380" (UID: "457d8563-627c-4188-8927-9cb145661380"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:32:30.721844 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.721754    5639 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/457d8563-627c-4188-8927-9cb145661380-metrics-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:30.721844 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.721785    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/457d8563-627c-4188-8927-9cb145661380-maas-api-tls\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:30.721844 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.721795    5639 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/457d8563-627c-4188-8927-9cb145661380-openshift-service-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:30.721844 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.721805    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-682gm\" (UniqueName: \"kubernetes.io/projected/457d8563-627c-4188-8927-9cb145661380-kube-api-access-682gm\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:30.907100 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.907065    5639 generic.go:356] "Generic (PLEG): container finished" podID="bb863a90-ffbe-46fe-b167-88efe18365ea" containerID="99a88161a850fe6c0cfe087a3812219c756cecbf3a4b185382cda76977790f12" exitCode=0
Nov 02 02:32:30.907592 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.907121    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp"
Nov 02 02:32:30.907592 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.907152    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp" event={"ID":"bb863a90-ffbe-46fe-b167-88efe18365ea","Type":"ContainerDied","Data":"99a88161a850fe6c0cfe087a3812219c756cecbf3a4b185382cda76977790f12"}
Nov 02 02:32:30.907592 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.907206    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp" event={"ID":"bb863a90-ffbe-46fe-b167-88efe18365ea","Type":"ContainerDied","Data":"e9693ccdb6f2d362d56d865efe5e9ccbb77bf4e0c294773190d0d3b5f5871323"}
Nov 02 02:32:30.907592 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.907236    5639 scope.go:122] "RemoveContainer" containerID="99a88161a850fe6c0cfe087a3812219c756cecbf3a4b185382cda76977790f12"
Nov 02 02:32:30.908652 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.908624    5639 generic.go:356] "Generic (PLEG): container finished" podID="457d8563-627c-4188-8927-9cb145661380" containerID="f4f5c74ed3422a1d6eec1b17293c3bdf880bcb2dab982466e15cee595ade1f96" exitCode=0
Nov 02 02:32:30.908781 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.908766    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m"
Nov 02 02:32:30.908864 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.908816    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m" event={"ID":"457d8563-627c-4188-8927-9cb145661380","Type":"ContainerDied","Data":"f4f5c74ed3422a1d6eec1b17293c3bdf880bcb2dab982466e15cee595ade1f96"}
Nov 02 02:32:30.908864 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.908858    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m" event={"ID":"457d8563-627c-4188-8927-9cb145661380","Type":"ContainerDied","Data":"af51cb02dc5c04afa72da717269acfb743148fd7955cb24f266f0b361bbbf8d9"}
Nov 02 02:32:30.910431 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.910387    5639 generic.go:356] "Generic (PLEG): container finished" podID="3b683b83-cd90-4840-87d6-158c702b9c50" containerID="4235135d7e5eae97645dc37cc55ad29ac1ad62bd8c4cc8dedcfda4cf38886d70" exitCode=0
Nov 02 02:32:30.910566 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.910442    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m" event={"ID":"3b683b83-cd90-4840-87d6-158c702b9c50","Type":"ContainerDied","Data":"4235135d7e5eae97645dc37cc55ad29ac1ad62bd8c4cc8dedcfda4cf38886d70"}
Nov 02 02:32:30.910566 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.910470    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m"
Nov 02 02:32:30.910566 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.910475    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m" event={"ID":"3b683b83-cd90-4840-87d6-158c702b9c50","Type":"ContainerDied","Data":"50f47124bd70eff21e62c6b2c17f713c186d181d887f2a272720c025f59e1f98"}
Nov 02 02:32:30.920582 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.919418    5639 scope.go:122] "RemoveContainer" containerID="99a88161a850fe6c0cfe087a3812219c756cecbf3a4b185382cda76977790f12"
Nov 02 02:32:30.920582 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:32:30.919720    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99a88161a850fe6c0cfe087a3812219c756cecbf3a4b185382cda76977790f12\": container with ID starting with 99a88161a850fe6c0cfe087a3812219c756cecbf3a4b185382cda76977790f12 not found: ID does not exist" containerID="99a88161a850fe6c0cfe087a3812219c756cecbf3a4b185382cda76977790f12"
Nov 02 02:32:30.920582 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.919760    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a88161a850fe6c0cfe087a3812219c756cecbf3a4b185382cda76977790f12"} err="failed to get container status \"99a88161a850fe6c0cfe087a3812219c756cecbf3a4b185382cda76977790f12\": rpc error: code = NotFound desc = could not find container \"99a88161a850fe6c0cfe087a3812219c756cecbf3a4b185382cda76977790f12\": container with ID starting with 99a88161a850fe6c0cfe087a3812219c756cecbf3a4b185382cda76977790f12 not found: ID does not exist"
Nov 02 02:32:30.920582 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.919785    5639 scope.go:122] "RemoveContainer" containerID="f4f5c74ed3422a1d6eec1b17293c3bdf880bcb2dab982466e15cee595ade1f96"
Nov 02 02:32:30.930127 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.930110    5639 scope.go:122] "RemoveContainer" containerID="f4f5c74ed3422a1d6eec1b17293c3bdf880bcb2dab982466e15cee595ade1f96"
Nov 02 02:32:30.930465 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:32:30.930446    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4f5c74ed3422a1d6eec1b17293c3bdf880bcb2dab982466e15cee595ade1f96\": container with ID starting with f4f5c74ed3422a1d6eec1b17293c3bdf880bcb2dab982466e15cee595ade1f96 not found: ID does not exist" containerID="f4f5c74ed3422a1d6eec1b17293c3bdf880bcb2dab982466e15cee595ade1f96"
Nov 02 02:32:30.930510 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.930473    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4f5c74ed3422a1d6eec1b17293c3bdf880bcb2dab982466e15cee595ade1f96"} err="failed to get container status \"f4f5c74ed3422a1d6eec1b17293c3bdf880bcb2dab982466e15cee595ade1f96\": rpc error: code = NotFound desc = could not find container \"f4f5c74ed3422a1d6eec1b17293c3bdf880bcb2dab982466e15cee595ade1f96\": container with ID starting with f4f5c74ed3422a1d6eec1b17293c3bdf880bcb2dab982466e15cee595ade1f96 not found: ID does not exist"
Nov 02 02:32:30.930510 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.930491    5639 scope.go:122] "RemoveContainer" containerID="4235135d7e5eae97645dc37cc55ad29ac1ad62bd8c4cc8dedcfda4cf38886d70"
Nov 02 02:32:30.938936 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.938918    5639 scope.go:122] "RemoveContainer" containerID="4235135d7e5eae97645dc37cc55ad29ac1ad62bd8c4cc8dedcfda4cf38886d70"
Nov 02 02:32:30.939195 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:32:30.939176    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4235135d7e5eae97645dc37cc55ad29ac1ad62bd8c4cc8dedcfda4cf38886d70\": container with ID starting with 4235135d7e5eae97645dc37cc55ad29ac1ad62bd8c4cc8dedcfda4cf38886d70 not found: ID does not exist" containerID="4235135d7e5eae97645dc37cc55ad29ac1ad62bd8c4cc8dedcfda4cf38886d70"
Nov 02 02:32:30.939270 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:30.939206    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4235135d7e5eae97645dc37cc55ad29ac1ad62bd8c4cc8dedcfda4cf38886d70"} err="failed to get container status \"4235135d7e5eae97645dc37cc55ad29ac1ad62bd8c4cc8dedcfda4cf38886d70\": rpc error: code = NotFound desc = could not find container \"4235135d7e5eae97645dc37cc55ad29ac1ad62bd8c4cc8dedcfda4cf38886d70\": container with ID starting with 4235135d7e5eae97645dc37cc55ad29ac1ad62bd8c4cc8dedcfda4cf38886d70 not found: ID does not exist"
Nov 02 02:32:31.070231 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:31.070133    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m"]
Nov 02 02:32:31.074093 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:31.074062    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9ab719a4-8697f98647-7qp7m"]
Nov 02 02:32:31.188484 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:31.188429    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m"]
Nov 02 02:32:31.191057 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:31.190983    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9ab719a4-5fb78cb5ff-8hs9m"]
Nov 02 02:32:31.217245 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:31.217210    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b683b83-cd90-4840-87d6-158c702b9c50" path="/var/lib/kubelet/pods/3b683b83-cd90-4840-87d6-158c702b9c50/volumes"
Nov 02 02:32:31.217651 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:31.217638    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="457d8563-627c-4188-8927-9cb145661380" path="/var/lib/kubelet/pods/457d8563-627c-4188-8927-9cb145661380/volumes"
Nov 02 02:32:31.298553 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:31.298515    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp"]
Nov 02 02:32:31.302586 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:31.302562    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9ab719a4-6b758b8b79-mb8xp"]
Nov 02 02:32:32.352089 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:32.352048    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg"]
Nov 02 02:32:32.402116 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:32.402083    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ab719a4-p2pzg"]
Nov 02 02:32:33.219731 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:33.219690    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e" path="/var/lib/kubelet/pods/a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e/volumes"
Nov 02 02:32:33.220146 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:33.220130    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb863a90-ffbe-46fe-b167-88efe18365ea" path="/var/lib/kubelet/pods/bb863a90-ffbe-46fe-b167-88efe18365ea/volumes"
Nov 02 02:32:34.471967 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:34.471934    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"]
Nov 02 02:32:34.472423 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:34.472155    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt" podUID="52bbf788-40fb-43df-90b8-2d9074e07ce3" containerName="istio-proxy" containerID="cri-o://d3a989734adc398615a7d1f4eaee3410f5d58b0d4a48131bb09eb438a480d5e8" gracePeriod=30
Nov 02 02:32:34.805235 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:34.805141    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:32:34.972736 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:34.972698    5639 patch_prober.go:29] interesting pod/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:32:34.972901 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:34.972766    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt" podUID="52bbf788-40fb-43df-90b8-2d9074e07ce3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:32:36.308858 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.308825    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"]
Nov 02 02:32:36.309335 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.309295    5639 cpu_manager.go:414] "containerMap: removing container" podUID="3b683b83-cd90-4840-87d6-158c702b9c50" containerName="payload-pre-processing"
Nov 02 02:32:36.309335 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.309311    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b683b83-cd90-4840-87d6-158c702b9c50" containerName="payload-pre-processing"
Nov 02 02:32:36.309335 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.309326    5639 cpu_manager.go:414] "containerMap: removing container" podUID="bb863a90-ffbe-46fe-b167-88efe18365ea" containerName="payload-processing"
Nov 02 02:32:36.309335 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.309331    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb863a90-ffbe-46fe-b167-88efe18365ea" containerName="payload-processing"
Nov 02 02:32:36.309494 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.309342    5639 cpu_manager.go:414] "containerMap: removing container" podUID="457d8563-627c-4188-8927-9cb145661380" containerName="maas-api"
Nov 02 02:32:36.309494 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.309346    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="457d8563-627c-4188-8927-9cb145661380" containerName="maas-api"
Nov 02 02:32:36.309494 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.309352    5639 cpu_manager.go:414] "containerMap: removing container" podUID="a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e" containerName="revoke-keys"
Nov 02 02:32:36.309494 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.309356    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e" containerName="revoke-keys"
Nov 02 02:32:36.309494 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.309456    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b683b83-cd90-4840-87d6-158c702b9c50" containerName="payload-pre-processing"
Nov 02 02:32:36.309494 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.309462    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb863a90-ffbe-46fe-b167-88efe18365ea" containerName="payload-processing"
Nov 02 02:32:36.309494 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.309469    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="457d8563-627c-4188-8927-9cb145661380" containerName="maas-api"
Nov 02 02:32:36.309494 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.309474    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6fbbbfd-6df9-47fa-b6d9-03f5929e9e6e" containerName="revoke-keys"
Nov 02 02:32:36.314351 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.314317    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.372145 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.371705    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-5f6a0b-openshift-default-dockercfg-mv2lv\""
Nov 02 02:32:36.377708 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.377579    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-data\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.377708 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.377638    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-envoy\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.377708 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.377670    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istiod-ca-cert\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.377708 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.377704    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-workload-certs\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.378108 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.377758    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4cqb\" (UniqueName: \"kubernetes.io/projected/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-kube-api-access-d4cqb\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.378108 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.377801    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-credential-socket\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.378108 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.377847    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-token\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.378108 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.377875    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-podinfo\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.378108 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.377944    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-workload-socket\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.423428 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.423394    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"]
Nov 02 02:32:36.479045 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.479009    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-data\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.479253 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.479051    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-envoy\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.479253 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.479187    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istiod-ca-cert\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.479253 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.479241    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-workload-certs\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.479555 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.479384    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4cqb\" (UniqueName: \"kubernetes.io/projected/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-kube-api-access-d4cqb\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.479555 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.479436    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-credential-socket\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.479555 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.479515    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-data\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.479555 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.479530    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-token\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.479812 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.479561    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-podinfo\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.479812 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.479635    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-workload-socket\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.480052 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.480030    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-workload-socket\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.480143 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.480055    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istiod-ca-cert\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.480143 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.480087    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-workload-certs\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.480451 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.480428    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-credential-socket\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.482599 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.482567    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-envoy\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.482898 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.482873    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-podinfo\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.540513 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.540472    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-token\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.540678 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.540591    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4cqb\" (UniqueName: \"kubernetes.io/projected/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-kube-api-access-d4cqb\") pod \"e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") " pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:36.627360 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:36.627308    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:37.057636 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:37.057609    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"]
Nov 02 02:32:37.059851 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:32:37.059820    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff6e2ea_c87e_411d_b3a9_1a17fc12d8c6.slice/crio-e7a7a0eb12685c9addf0c14d49ed09fd2e33a72e86db41841e08705ab0a06c20 WatchSource:0}: Error finding container e7a7a0eb12685c9addf0c14d49ed09fd2e33a72e86db41841e08705ab0a06c20: Status 404 returned error can't find the container with id e7a7a0eb12685c9addf0c14d49ed09fd2e33a72e86db41841e08705ab0a06c20
Nov 02 02:32:37.062112 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:37.062080    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:32:37.062193 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:37.062149    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:32:37.062193 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:37.062178    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:32:37.953327 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:37.953263    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4" event={"ID":"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6","Type":"ContainerStarted","Data":"22599a1fcc32b808a6f42b765acbed80f8b8a48ebc874e5850e7de44d4d8bd55"}
Nov 02 02:32:37.953772 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:37.953373    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4" event={"ID":"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6","Type":"ContainerStarted","Data":"e7a7a0eb12685c9addf0c14d49ed09fd2e33a72e86db41841e08705ab0a06c20"}
Nov 02 02:32:38.123015 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:38.122950    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4" podStartSLOduration=2.122933855 podStartE2EDuration="2.122933855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:32:38.122088684 +0000 UTC m=+4023.025770792" watchObservedRunningTime="2026-11-02 02:32:38.122933855 +0000 UTC m=+4023.026615969"
Nov 02 02:32:38.627971 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:38.627932    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:38.633327 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:38.633274    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:38.959579 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:38.959498    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:38.960586 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:38.960566    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:32:39.732593 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.732568    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:32:39.817360 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.817255    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-credential-socket\") pod \"52bbf788-40fb-43df-90b8-2d9074e07ce3\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") "
Nov 02 02:32:39.817360 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.817339    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52bbf788-40fb-43df-90b8-2d9074e07ce3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bbf788-40fb-43df-90b8-2d9074e07ce3-istiod-ca-cert\") pod \"52bbf788-40fb-43df-90b8-2d9074e07ce3\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") "
Nov 02 02:32:39.817584 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.817363    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-token\" (UniqueName: \"kubernetes.io/projected/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-token\") pod \"52bbf788-40fb-43df-90b8-2d9074e07ce3\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") "
Nov 02 02:32:39.817584 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.817389    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52bbf788-40fb-43df-90b8-2d9074e07ce3-kube-api-access-mcdcz\" (UniqueName: \"kubernetes.io/projected/52bbf788-40fb-43df-90b8-2d9074e07ce3-kube-api-access-mcdcz\") pod \"52bbf788-40fb-43df-90b8-2d9074e07ce3\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") "
Nov 02 02:32:39.817584 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.817424    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-workload-certs\") pod \"52bbf788-40fb-43df-90b8-2d9074e07ce3\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") "
Nov 02 02:32:39.817749 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.817561    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-credential-socket" pod "52bbf788-40fb-43df-90b8-2d9074e07ce3" (UID: "52bbf788-40fb-43df-90b8-2d9074e07ce3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:32:39.817749 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.817593    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-envoy\") pod \"52bbf788-40fb-43df-90b8-2d9074e07ce3\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") "
Nov 02 02:32:39.817749 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.817659    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-podinfo\") pod \"52bbf788-40fb-43df-90b8-2d9074e07ce3\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") "
Nov 02 02:32:39.817749 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.817706    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-data\") pod \"52bbf788-40fb-43df-90b8-2d9074e07ce3\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") "
Nov 02 02:32:39.817983 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.817753    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52bbf788-40fb-43df-90b8-2d9074e07ce3-istiod-ca-cert" pod "52bbf788-40fb-43df-90b8-2d9074e07ce3" (UID: "52bbf788-40fb-43df-90b8-2d9074e07ce3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:32:39.817983 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.817758    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-workload-socket\") pod \"52bbf788-40fb-43df-90b8-2d9074e07ce3\" (UID: \"52bbf788-40fb-43df-90b8-2d9074e07ce3\") "
Nov 02 02:32:39.817983 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.817822    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-workload-certs" pod "52bbf788-40fb-43df-90b8-2d9074e07ce3" (UID: "52bbf788-40fb-43df-90b8-2d9074e07ce3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:32:39.818144 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.817990    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-data" pod "52bbf788-40fb-43df-90b8-2d9074e07ce3" (UID: "52bbf788-40fb-43df-90b8-2d9074e07ce3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:32:39.818144 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.818077    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-workload-socket" pod "52bbf788-40fb-43df-90b8-2d9074e07ce3" (UID: "52bbf788-40fb-43df-90b8-2d9074e07ce3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:32:39.818222 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.818198    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:39.818222 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.818214    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:39.818314 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.818224    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:39.818314 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.818232    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bbf788-40fb-43df-90b8-2d9074e07ce3-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:39.818314 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.818241    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:39.820261 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.820235    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-envoy" pod "52bbf788-40fb-43df-90b8-2d9074e07ce3" (UID: "52bbf788-40fb-43df-90b8-2d9074e07ce3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:32:39.820387 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.820274    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52bbf788-40fb-43df-90b8-2d9074e07ce3-kube-api-access-mcdcz" pod "52bbf788-40fb-43df-90b8-2d9074e07ce3" (UID: "52bbf788-40fb-43df-90b8-2d9074e07ce3"). InnerVolumeSpecName "kube-api-access-mcdcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:32:39.820387 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.820370    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-podinfo" pod "52bbf788-40fb-43df-90b8-2d9074e07ce3" (UID: "52bbf788-40fb-43df-90b8-2d9074e07ce3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:32:39.820514 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.820427    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-token" pod "52bbf788-40fb-43df-90b8-2d9074e07ce3" (UID: "52bbf788-40fb-43df-90b8-2d9074e07ce3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:32:39.919057 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.919017    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:39.919057 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.919046    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:39.919057 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.919057    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bbf788-40fb-43df-90b8-2d9074e07ce3-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:39.919724 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.919072    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcdcz\" (UniqueName: \"kubernetes.io/projected/52bbf788-40fb-43df-90b8-2d9074e07ce3-kube-api-access-mcdcz\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:39.967894 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.967861    5639 generic.go:356] "Generic (PLEG): container finished" podID="52bbf788-40fb-43df-90b8-2d9074e07ce3" containerID="d3a989734adc398615a7d1f4eaee3410f5d58b0d4a48131bb09eb438a480d5e8" exitCode=0
Nov 02 02:32:39.968362 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.967921    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"
Nov 02 02:32:39.968362 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.967944    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt" event={"ID":"52bbf788-40fb-43df-90b8-2d9074e07ce3","Type":"ContainerDied","Data":"d3a989734adc398615a7d1f4eaee3410f5d58b0d4a48131bb09eb438a480d5e8"}
Nov 02 02:32:39.968362 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.967990    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt" event={"ID":"52bbf788-40fb-43df-90b8-2d9074e07ce3","Type":"ContainerDied","Data":"8a6977856cf2e094ab3b03194ef5a32c8f394cb8ea23f124a687e5da6cb6e374"}
Nov 02 02:32:39.968362 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.968010    5639 scope.go:122] "RemoveContainer" containerID="d3a989734adc398615a7d1f4eaee3410f5d58b0d4a48131bb09eb438a480d5e8"
Nov 02 02:32:39.978455 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.978437    5639 scope.go:122] "RemoveContainer" containerID="3580cab30d99b54ce4f19b4738c2820b102071f7ac22d1040523670e9009d5b3"
Nov 02 02:32:39.987811 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.987792    5639 scope.go:122] "RemoveContainer" containerID="d3a989734adc398615a7d1f4eaee3410f5d58b0d4a48131bb09eb438a480d5e8"
Nov 02 02:32:39.988089 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:32:39.988065    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3a989734adc398615a7d1f4eaee3410f5d58b0d4a48131bb09eb438a480d5e8\": container with ID starting with d3a989734adc398615a7d1f4eaee3410f5d58b0d4a48131bb09eb438a480d5e8 not found: ID does not exist" containerID="d3a989734adc398615a7d1f4eaee3410f5d58b0d4a48131bb09eb438a480d5e8"
Nov 02 02:32:39.988170 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.988104    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3a989734adc398615a7d1f4eaee3410f5d58b0d4a48131bb09eb438a480d5e8"} err="failed to get container status \"d3a989734adc398615a7d1f4eaee3410f5d58b0d4a48131bb09eb438a480d5e8\": rpc error: code = NotFound desc = could not find container \"d3a989734adc398615a7d1f4eaee3410f5d58b0d4a48131bb09eb438a480d5e8\": container with ID starting with d3a989734adc398615a7d1f4eaee3410f5d58b0d4a48131bb09eb438a480d5e8 not found: ID does not exist"
Nov 02 02:32:39.988170 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.988130    5639 scope.go:122] "RemoveContainer" containerID="3580cab30d99b54ce4f19b4738c2820b102071f7ac22d1040523670e9009d5b3"
Nov 02 02:32:39.988437 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:32:39.988416    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3580cab30d99b54ce4f19b4738c2820b102071f7ac22d1040523670e9009d5b3\": container with ID starting with 3580cab30d99b54ce4f19b4738c2820b102071f7ac22d1040523670e9009d5b3 not found: ID does not exist" containerID="3580cab30d99b54ce4f19b4738c2820b102071f7ac22d1040523670e9009d5b3"
Nov 02 02:32:39.988482 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:39.988444    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3580cab30d99b54ce4f19b4738c2820b102071f7ac22d1040523670e9009d5b3"} err="failed to get container status \"3580cab30d99b54ce4f19b4738c2820b102071f7ac22d1040523670e9009d5b3\": rpc error: code = NotFound desc = could not find container \"3580cab30d99b54ce4f19b4738c2820b102071f7ac22d1040523670e9009d5b3\": container with ID starting with 3580cab30d99b54ce4f19b4738c2820b102071f7ac22d1040523670e9009d5b3 not found: ID does not exist"
Nov 02 02:32:40.133155 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:40.133124    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"]
Nov 02 02:32:40.137622 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:40.137596    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9ab719a4-openshift-default-575f99b88f-nb6qt"]
Nov 02 02:32:40.271375 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:40.271342    5639 scope.go:122] "RemoveContainer" containerID="2f387a7ce05fddc603ed7bffe5d160654a6f337c1e7a4da7f8c603a51ec352be"
Nov 02 02:32:41.217845 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:41.217808    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52bbf788-40fb-43df-90b8-2d9074e07ce3" path="/var/lib/kubelet/pods/52bbf788-40fb-43df-90b8-2d9074e07ce3/volumes"
Nov 02 02:32:47.144026 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:47.143985    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh"]
Nov 02 02:32:47.144771 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:47.144699    5639 cpu_manager.go:414] "containerMap: removing container" podUID="52bbf788-40fb-43df-90b8-2d9074e07ce3" containerName="istio-proxy"
Nov 02 02:32:47.144771 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:47.144721    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52bbf788-40fb-43df-90b8-2d9074e07ce3" containerName="istio-proxy"
Nov 02 02:32:47.144771 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:47.144760    5639 cpu_manager.go:414] "containerMap: removing container" podUID="52bbf788-40fb-43df-90b8-2d9074e07ce3" containerName="istio-proxy"
Nov 02 02:32:47.144771 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:47.144769    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52bbf788-40fb-43df-90b8-2d9074e07ce3" containerName="istio-proxy"
Nov 02 02:32:47.145013 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:47.144869    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="52bbf788-40fb-43df-90b8-2d9074e07ce3" containerName="istio-proxy"
Nov 02 02:32:47.148218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:47.148196    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh"
Nov 02 02:32:47.200557 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:47.200495    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-5f6a0b-serving-cert\""
Nov 02 02:32:47.200557 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:47.200501    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:32:47.250148 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:47.250113    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:32:47.250380 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:47.250173    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nk57n\""
Nov 02 02:32:47.250380 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:47.250177    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 02:32:47.252370 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:47.252351    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh"]
Nov 02 02:32:47.297358 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:47.297308    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6605af90-6fb9-4225-abd8-5af5de1bbfd1-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh\" (UID: \"6605af90-6fb9-4225-abd8-5af5de1bbfd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh"
Nov 02 02:32:47.297578 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:47.297463    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgrbm\" (UniqueName: \"kubernetes.io/projected/6605af90-6fb9-4225-abd8-5af5de1bbfd1-kube-api-access-fgrbm\") pod \"maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh\" (UID: \"6605af90-6fb9-4225-abd8-5af5de1bbfd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh"
Nov 02 02:32:47.297663 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:47.297580    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6605af90-6fb9-4225-abd8-5af5de1bbfd1-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh\" (UID: \"6605af90-6fb9-4225-abd8-5af5de1bbfd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh"
Nov 02 02:32:47.297726 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:47.297691    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6605af90-6fb9-4225-abd8-5af5de1bbfd1-metrics-certs\") pod \"maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh\" (UID: \"6605af90-6fb9-4225-abd8-5af5de1bbfd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh"
Nov 02 02:32:47.399530 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:47.399426    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6605af90-6fb9-4225-abd8-5af5de1bbfd1-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh\" (UID: \"6605af90-6fb9-4225-abd8-5af5de1bbfd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh"
Nov 02 02:32:47.399530 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:47.399513    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgrbm\" (UniqueName: \"kubernetes.io/projected/6605af90-6fb9-4225-abd8-5af5de1bbfd1-kube-api-access-fgrbm\") pod \"maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh\" (UID: \"6605af90-6fb9-4225-abd8-5af5de1bbfd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh"
Nov 02 02:32:47.399785 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:47.399565    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6605af90-6fb9-4225-abd8-5af5de1bbfd1-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh\" (UID: \"6605af90-6fb9-4225-abd8-5af5de1bbfd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh"
Nov 02 02:32:47.399785 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:32:47.399607    5639 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-serving-cert: secret "maas-api-e2e-authcfg-http-5f6a0b-serving-cert" not found
Nov 02 02:32:47.399958 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:32:47.399909    5639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6605af90-6fb9-4225-abd8-5af5de1bbfd1-maas-api-tls podName:6605af90-6fb9-4225-abd8-5af5de1bbfd1 nodeName:}" failed. No retries permitted until 2026-11-02 02:32:47.899733305 +0000 UTC m=+4032.803415415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6605af90-6fb9-4225-abd8-5af5de1bbfd1-maas-api-tls") pod "maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh" (UID: "6605af90-6fb9-4225-abd8-5af5de1bbfd1") : secret "maas-api-e2e-authcfg-http-5f6a0b-serving-cert" not found
Nov 02 02:32:47.400943 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:47.400915    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6605af90-6fb9-4225-abd8-5af5de1bbfd1-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh\" (UID: \"6605af90-6fb9-4225-abd8-5af5de1bbfd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh"
Nov 02 02:32:47.401063 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:47.399619    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6605af90-6fb9-4225-abd8-5af5de1bbfd1-metrics-certs\") pod \"maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh\" (UID: \"6605af90-6fb9-4225-abd8-5af5de1bbfd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh"
Nov 02 02:32:47.402746 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:47.402720    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6605af90-6fb9-4225-abd8-5af5de1bbfd1-metrics-certs\") pod \"maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh\" (UID: \"6605af90-6fb9-4225-abd8-5af5de1bbfd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh"
Nov 02 02:32:47.464534 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:47.464497    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgrbm\" (UniqueName: \"kubernetes.io/projected/6605af90-6fb9-4225-abd8-5af5de1bbfd1-kube-api-access-fgrbm\") pod \"maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh\" (UID: \"6605af90-6fb9-4225-abd8-5af5de1bbfd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh"
Nov 02 02:32:47.906056 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:47.906021    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6605af90-6fb9-4225-abd8-5af5de1bbfd1-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh\" (UID: \"6605af90-6fb9-4225-abd8-5af5de1bbfd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh"
Nov 02 02:32:47.908705 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:47.908679    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6605af90-6fb9-4225-abd8-5af5de1bbfd1-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh\" (UID: \"6605af90-6fb9-4225-abd8-5af5de1bbfd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh"
Nov 02 02:32:48.061374 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:48.061343    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh"
Nov 02 02:32:48.436343 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:48.436321    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh"]
Nov 02 02:32:48.438528 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:32:48.438495    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6605af90_6fb9_4225_abd8_5af5de1bbfd1.slice/crio-d2186209e93092d82ba4b70f6a085899e8508a87dd055f9d49fcaa938b8f71c8 WatchSource:0}: Error finding container d2186209e93092d82ba4b70f6a085899e8508a87dd055f9d49fcaa938b8f71c8: Status 404 returned error can't find the container with id d2186209e93092d82ba4b70f6a085899e8508a87dd055f9d49fcaa938b8f71c8
Nov 02 02:32:49.013243 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:49.013206    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh" event={"ID":"6605af90-6fb9-4225-abd8-5af5de1bbfd1","Type":"ContainerStarted","Data":"d2186209e93092d82ba4b70f6a085899e8508a87dd055f9d49fcaa938b8f71c8"}
Nov 02 02:32:50.020007 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:50.019963    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh" event={"ID":"6605af90-6fb9-4225-abd8-5af5de1bbfd1","Type":"ContainerStarted","Data":"463cc8139a901904ccceb350d75c7467a49498ba98e68ef182ff091aff88fec7"}
Nov 02 02:32:50.020481 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:50.020083    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh"
Nov 02 02:32:50.137317 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:50.136986    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh" podStartSLOduration=2.248208203 podStartE2EDuration="3.136969889s" totalImagesPullingTime="888.761686ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:32:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:32:50.136610214 +0000 UTC m=+4035.040292329" watchObservedRunningTime="2026-11-02 02:32:50.136969889 +0000 UTC m=+4035.040652005"
Nov 02 02:32:53.428387 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:53.428347    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz"]
Nov 02 02:32:53.429307 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:53.429243    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="52bbf788-40fb-43df-90b8-2d9074e07ce3" containerName="istio-proxy"
Nov 02 02:32:53.436005 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:53.435967    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz"
Nov 02 02:32:53.488672 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:53.488636    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz"]
Nov 02 02:32:53.535584 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:53.535545    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-5f6a0b\""
Nov 02 02:32:53.535584 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:53.535576    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-5f6a0b-dockercfg-f47rn\""
Nov 02 02:32:53.579378 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:53.579326    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgwg6\" (UniqueName: \"kubernetes.io/projected/40b88690-e740-4b6c-9edc-bdd3ddfa6092-kube-api-access-zgwg6\") pod \"payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz\" (UID: \"40b88690-e740-4b6c-9edc-bdd3ddfa6092\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz"
Nov 02 02:32:53.579378 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:53.579388    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b88690-e740-4b6c-9edc-bdd3ddfa6092-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz\" (UID: \"40b88690-e740-4b6c-9edc-bdd3ddfa6092\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz"
Nov 02 02:32:53.599602 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:53.599522    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg"]
Nov 02 02:32:53.603974 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:53.603951    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg"
Nov 02 02:32:53.660709 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:53.660673    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg"]
Nov 02 02:32:53.680495 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:53.680401    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgwg6\" (UniqueName: \"kubernetes.io/projected/40b88690-e740-4b6c-9edc-bdd3ddfa6092-kube-api-access-zgwg6\") pod \"payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz\" (UID: \"40b88690-e740-4b6c-9edc-bdd3ddfa6092\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz"
Nov 02 02:32:53.680495 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:53.680448    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b88690-e740-4b6c-9edc-bdd3ddfa6092-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz\" (UID: \"40b88690-e740-4b6c-9edc-bdd3ddfa6092\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz"
Nov 02 02:32:53.681230 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:53.681206    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b88690-e740-4b6c-9edc-bdd3ddfa6092-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz\" (UID: \"40b88690-e740-4b6c-9edc-bdd3ddfa6092\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz"
Nov 02 02:32:53.741435 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:53.741402    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgwg6\" (UniqueName: \"kubernetes.io/projected/40b88690-e740-4b6c-9edc-bdd3ddfa6092-kube-api-access-zgwg6\") pod \"payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz\" (UID: \"40b88690-e740-4b6c-9edc-bdd3ddfa6092\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz"
Nov 02 02:32:53.752248 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:53.752220    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz"
Nov 02 02:32:53.782011 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:53.781975    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt9jp\" (UniqueName: \"kubernetes.io/projected/79951504-4132-4161-b332-fc7418432ed2-kube-api-access-gt9jp\") pod \"payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg\" (UID: \"79951504-4132-4161-b332-fc7418432ed2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg"
Nov 02 02:32:53.782196 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:53.782130    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79951504-4132-4161-b332-fc7418432ed2-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg\" (UID: \"79951504-4132-4161-b332-fc7418432ed2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg"
Nov 02 02:32:53.883268 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:53.883224    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt9jp\" (UniqueName: \"kubernetes.io/projected/79951504-4132-4161-b332-fc7418432ed2-kube-api-access-gt9jp\") pod \"payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg\" (UID: \"79951504-4132-4161-b332-fc7418432ed2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg"
Nov 02 02:32:53.883471 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:53.883338    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79951504-4132-4161-b332-fc7418432ed2-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg\" (UID: \"79951504-4132-4161-b332-fc7418432ed2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg"
Nov 02 02:32:53.884069 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:53.884046    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79951504-4132-4161-b332-fc7418432ed2-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg\" (UID: \"79951504-4132-4161-b332-fc7418432ed2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg"
Nov 02 02:32:53.944961 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:53.944878    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt9jp\" (UniqueName: \"kubernetes.io/projected/79951504-4132-4161-b332-fc7418432ed2-kube-api-access-gt9jp\") pod \"payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg\" (UID: \"79951504-4132-4161-b332-fc7418432ed2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg"
Nov 02 02:32:54.126700 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:54.126673    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz"]
Nov 02 02:32:54.128989 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:32:54.128959    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b88690_e740_4b6c_9edc_bdd3ddfa6092.slice/crio-63f7b1965c61e76d2b880d00d4da12e3d952a2f0ffa44ce2cef1bceea3d3c14c WatchSource:0}: Error finding container 63f7b1965c61e76d2b880d00d4da12e3d952a2f0ffa44ce2cef1bceea3d3c14c: Status 404 returned error can't find the container with id 63f7b1965c61e76d2b880d00d4da12e3d952a2f0ffa44ce2cef1bceea3d3c14c
Nov 02 02:32:54.217730 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:54.217639    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg"
Nov 02 02:32:54.603727 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:54.603702    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg"]
Nov 02 02:32:54.605954 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:32:54.605924    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79951504_4132_4161_b332_fc7418432ed2.slice/crio-cdb1485f3b56dadbc104cd7d223dc1b8e1c62499ac61bbe180929ca5c78d679a WatchSource:0}: Error finding container cdb1485f3b56dadbc104cd7d223dc1b8e1c62499ac61bbe180929ca5c78d679a: Status 404 returned error can't find the container with id cdb1485f3b56dadbc104cd7d223dc1b8e1c62499ac61bbe180929ca5c78d679a
Nov 02 02:32:55.047768 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:55.047664    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz" event={"ID":"40b88690-e740-4b6c-9edc-bdd3ddfa6092","Type":"ContainerStarted","Data":"f0d41d435ad60f0068740a2a47594205eddec7331ab0334dd8780485364d7d43"}
Nov 02 02:32:55.047768 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:55.047735    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz"
Nov 02 02:32:55.047768 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:55.047760    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz" event={"ID":"40b88690-e740-4b6c-9edc-bdd3ddfa6092","Type":"ContainerStarted","Data":"63f7b1965c61e76d2b880d00d4da12e3d952a2f0ffa44ce2cef1bceea3d3c14c"}
Nov 02 02:32:55.049481 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:55.049451    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg" event={"ID":"79951504-4132-4161-b332-fc7418432ed2","Type":"ContainerStarted","Data":"ce1ff7b5d6a9733200712602a3137ea6d3ca36aae9adb473e810295234711469"}
Nov 02 02:32:55.049641 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:55.049497    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg" event={"ID":"79951504-4132-4161-b332-fc7418432ed2","Type":"ContainerStarted","Data":"cdb1485f3b56dadbc104cd7d223dc1b8e1c62499ac61bbe180929ca5c78d679a"}
Nov 02 02:32:55.049641 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:55.049555    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg"
Nov 02 02:32:55.164720 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:55.164666    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz" podStartSLOduration=2.164651397 podStartE2EDuration="2.164651397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:32:55.163881501 +0000 UTC m=+4040.067563615" watchObservedRunningTime="2026-11-02 02:32:55.164651397 +0000 UTC m=+4040.068333510"
Nov 02 02:32:55.281052 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:55.280984    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg" podStartSLOduration=2.280966296 podStartE2EDuration="2.280966296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:32:55.279082892 +0000 UTC m=+4040.182765007" watchObservedRunningTime="2026-11-02 02:32:55.280966296 +0000 UTC m=+4040.184648464"
Nov 02 02:32:55.312406 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:55.312316    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:32:56.031329 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:32:56.031301    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh"
Nov 02 02:32:57.421629 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:32:57.421588    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:33:05.131193 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:05.131158    5639 generic.go:356] "Generic (PLEG): container finished" podID="dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6" containerID="22599a1fcc32b808a6f42b765acbed80f8b8a48ebc874e5850e7de44d4d8bd55" exitCode=0
Nov 02 02:33:05.131680 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:05.131232    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4" event={"ID":"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6","Type":"ContainerDied","Data":"22599a1fcc32b808a6f42b765acbed80f8b8a48ebc874e5850e7de44d4d8bd55"}
Nov 02 02:33:05.131680 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:05.131658    5639 scope.go:122] "RemoveContainer" containerID="22599a1fcc32b808a6f42b765acbed80f8b8a48ebc874e5850e7de44d4d8bd55"
Nov 02 02:33:05.133012 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:05.132980    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:33:05.133088 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:05.133071    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:33:05.133127 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:05.133104    5639 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 02:33:06.058430 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:06.058395    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz"
Nov 02 02:33:06.058624 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:06.058581    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg"
Nov 02 02:33:06.138595 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:06.138559    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4" event={"ID":"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6","Type":"ContainerStarted","Data":"5b7c02f9d5fb095af75ec3b6580f15ba060b312be9c5babcf4ab353198757b4c"}
Nov 02 02:33:06.628136 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:06.628095    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:33:06.629339 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:06.629313    5639 patch_prober.go:29] interesting pod/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.61:15021/healthz/ready\": dial tcp 10.134.1.61:15021: connect: connection refused" start-of-body=
Nov 02 02:33:06.629475 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:06.629364    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4" podUID="dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.61:15021/healthz/ready\": dial tcp 10.134.1.61:15021: connect: connection refused"
Nov 02 02:33:07.628100 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:07.628067    5639 patch_prober.go:29] interesting pod/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.61:15021/healthz/ready\": dial tcp 10.134.1.61:15021: connect: connection refused" start-of-body=
Nov 02 02:33:07.628554 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:07.628135    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4" podUID="dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.61:15021/healthz/ready\": dial tcp 10.134.1.61:15021: connect: connection refused"
Nov 02 02:33:08.628511 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:08.628469    5639 patch_prober.go:29] interesting pod/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.61:15021/healthz/ready\": dial tcp 10.134.1.61:15021: connect: connection refused" start-of-body=
Nov 02 02:33:08.628930 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:08.628524    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4" podUID="dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.61:15021/healthz/ready\": dial tcp 10.134.1.61:15021: connect: connection refused"
Nov 02 02:33:08.970415 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:08.965396    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:33:09.628566 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:09.628520    5639 patch_prober.go:29] interesting pod/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.61:15021/healthz/ready\": dial tcp 10.134.1.61:15021: connect: connection refused" start-of-body=
Nov 02 02:33:09.628986 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:09.628577    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4" podUID="dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.61:15021/healthz/ready\": dial tcp 10.134.1.61:15021: connect: connection refused"
Nov 02 02:33:10.627906 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:10.627866    5639 patch_prober.go:29] interesting pod/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.61:15021/healthz/ready\": dial tcp 10.134.1.61:15021: connect: connection refused" start-of-body=
Nov 02 02:33:10.628118 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:10.627921    5639 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4" podUID="dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.61:15021/healthz/ready\": dial tcp 10.134.1.61:15021: connect: connection refused"
Nov 02 02:33:11.632707 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:11.632668    5639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:33:11.633951 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:11.633934    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:33:27.421597 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:33:27.421507    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:33:29.046926 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:29.046887    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d63dc943-ft5fl"]
Nov 02 02:33:29.096544 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:29.096502    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d63dc943-ft5fl"]
Nov 02 02:33:29.217811 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:29.217774    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b3706cf-d187-4c88-bafd-c9c85738aeb5" path="/var/lib/kubelet/pods/8b3706cf-d187-4c88-bafd-c9c85738aeb5/volumes"
Nov 02 02:33:40.310320 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:33:40.310262    5639 scope.go:122] "RemoveContainer" containerID="b6ed07452fa68995f016207b835d983b64b0246243e93235d089a2c175ab48ef"
Nov 02 02:33:57.421779 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:33:57.421737    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:34:26.411931 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:26.411892    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb"]
Nov 02 02:34:26.415709 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:26.415691    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb"
Nov 02 02:34:26.478667 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:26.478633    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:34:26.480584 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:26.480558    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb"]
Nov 02 02:34:26.519999 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:26.519970    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhr9x\""
Nov 02 02:34:26.605075 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:26.605029    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82560d25-c97a-4bff-96cd-943f7746489f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb\" (UID: \"82560d25-c97a-4bff-96cd-943f7746489f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb"
Nov 02 02:34:26.605075 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:26.605076    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82560d25-c97a-4bff-96cd-943f7746489f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb\" (UID: \"82560d25-c97a-4bff-96cd-943f7746489f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb"
Nov 02 02:34:26.605362 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:26.605107    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghcg6\" (UniqueName: \"kubernetes.io/projected/82560d25-c97a-4bff-96cd-943f7746489f-kube-api-access-ghcg6\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb\" (UID: \"82560d25-c97a-4bff-96cd-943f7746489f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb"
Nov 02 02:34:26.706740 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:26.706641    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82560d25-c97a-4bff-96cd-943f7746489f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb\" (UID: \"82560d25-c97a-4bff-96cd-943f7746489f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb"
Nov 02 02:34:26.706740 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:26.706682    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82560d25-c97a-4bff-96cd-943f7746489f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb\" (UID: \"82560d25-c97a-4bff-96cd-943f7746489f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb"
Nov 02 02:34:26.706740 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:26.706704    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghcg6\" (UniqueName: \"kubernetes.io/projected/82560d25-c97a-4bff-96cd-943f7746489f-kube-api-access-ghcg6\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb\" (UID: \"82560d25-c97a-4bff-96cd-943f7746489f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb"
Nov 02 02:34:26.707372 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:26.707339    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82560d25-c97a-4bff-96cd-943f7746489f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb\" (UID: \"82560d25-c97a-4bff-96cd-943f7746489f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb"
Nov 02 02:34:26.709319 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:26.709299    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82560d25-c97a-4bff-96cd-943f7746489f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb\" (UID: \"82560d25-c97a-4bff-96cd-943f7746489f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb"
Nov 02 02:34:26.767543 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:26.767506    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghcg6\" (UniqueName: \"kubernetes.io/projected/82560d25-c97a-4bff-96cd-943f7746489f-kube-api-access-ghcg6\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb\" (UID: \"82560d25-c97a-4bff-96cd-943f7746489f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb"
Nov 02 02:34:27.028984 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:27.028892    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb"
Nov 02 02:34:27.401501 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:27.401474    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb"]
Nov 02 02:34:27.404515 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:34:27.404483    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82560d25_c97a_4bff_96cd_943f7746489f.slice/crio-15cbfe660ab7812b0ef218d850de5e76f2ba46132596fab21b5716c89ada1b7a WatchSource:0}: Error finding container 15cbfe660ab7812b0ef218d850de5e76f2ba46132596fab21b5716c89ada1b7a: Status 404 returned error can't find the container with id 15cbfe660ab7812b0ef218d850de5e76f2ba46132596fab21b5716c89ada1b7a
Nov 02 02:34:27.426640 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:34:27.426599    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:34:27.589062 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:27.589029    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb" event={"ID":"82560d25-c97a-4bff-96cd-943f7746489f","Type":"ContainerStarted","Data":"15cbfe660ab7812b0ef218d850de5e76f2ba46132596fab21b5716c89ada1b7a"}
Nov 02 02:34:28.595970 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:28.595935    5639 generic.go:356] "Generic (PLEG): container finished" podID="82560d25-c97a-4bff-96cd-943f7746489f" containerID="fae0764bce372d7ce970a002abaedaa78015b3aed03f8b27cabfa662856ca915" exitCode=0
Nov 02 02:34:28.596407 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:28.596028    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb" event={"ID":"82560d25-c97a-4bff-96cd-943f7746489f","Type":"ContainerDied","Data":"fae0764bce372d7ce970a002abaedaa78015b3aed03f8b27cabfa662856ca915"}
Nov 02 02:34:29.743196 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:29.743167    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb"
Nov 02 02:34:29.838496 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:29.838463    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82560d25-c97a-4bff-96cd-943f7746489f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82560d25-c97a-4bff-96cd-943f7746489f-maas-api-openshift-ca\") pod \"82560d25-c97a-4bff-96cd-943f7746489f\" (UID: \"82560d25-c97a-4bff-96cd-943f7746489f\") "
Nov 02 02:34:29.838697 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:29.838545    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82560d25-c97a-4bff-96cd-943f7746489f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82560d25-c97a-4bff-96cd-943f7746489f-maas-api-xks-ca\") pod \"82560d25-c97a-4bff-96cd-943f7746489f\" (UID: \"82560d25-c97a-4bff-96cd-943f7746489f\") "
Nov 02 02:34:29.838697 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:29.838645    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82560d25-c97a-4bff-96cd-943f7746489f-kube-api-access-ghcg6\" (UniqueName: \"kubernetes.io/projected/82560d25-c97a-4bff-96cd-943f7746489f-kube-api-access-ghcg6\") pod \"82560d25-c97a-4bff-96cd-943f7746489f\" (UID: \"82560d25-c97a-4bff-96cd-943f7746489f\") "
Nov 02 02:34:29.838919 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:29.838892    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82560d25-c97a-4bff-96cd-943f7746489f-maas-api-openshift-ca" pod "82560d25-c97a-4bff-96cd-943f7746489f" (UID: "82560d25-c97a-4bff-96cd-943f7746489f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:34:29.839053 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:29.839038    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82560d25-c97a-4bff-96cd-943f7746489f-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:34:29.840900 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:29.840875    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82560d25-c97a-4bff-96cd-943f7746489f-maas-api-xks-ca" pod "82560d25-c97a-4bff-96cd-943f7746489f" (UID: "82560d25-c97a-4bff-96cd-943f7746489f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:34:29.841018 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:29.840878    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82560d25-c97a-4bff-96cd-943f7746489f-kube-api-access-ghcg6" pod "82560d25-c97a-4bff-96cd-943f7746489f" (UID: "82560d25-c97a-4bff-96cd-943f7746489f"). InnerVolumeSpecName "kube-api-access-ghcg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:34:29.940524 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:29.940442    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82560d25-c97a-4bff-96cd-943f7746489f-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:34:29.940524 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:29.940471    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghcg6\" (UniqueName: \"kubernetes.io/projected/82560d25-c97a-4bff-96cd-943f7746489f-kube-api-access-ghcg6\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:34:30.607425 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:30.607389    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb"
Nov 02 02:34:30.607595 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:30.607391    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb" event={"ID":"82560d25-c97a-4bff-96cd-943f7746489f","Type":"ContainerDied","Data":"15cbfe660ab7812b0ef218d850de5e76f2ba46132596fab21b5716c89ada1b7a"}
Nov 02 02:34:30.607595 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:30.607523    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15cbfe660ab7812b0ef218d850de5e76f2ba46132596fab21b5716c89ada1b7a"
Nov 02 02:34:37.923208 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:37.923170    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz"]
Nov 02 02:34:38.029138 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:38.029093    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg"]
Nov 02 02:34:40.333034 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:40.333001    5639 scope.go:122] "RemoveContainer" containerID="1aeb02100fc2a57ea51da832262534b705a4c352e0371a24fdfd165e420ea04e"
Nov 02 02:34:40.342165 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:40.342141    5639 scope.go:122] "RemoveContainer" containerID="c1f2d3712f7c44e4b00f11a0041c14a6e28bdd4f7201ae2f932b1dc502e3f8ab"
Nov 02 02:34:42.924291 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:42.924230    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz" podUID="40b88690-e740-4b6c-9edc-bdd3ddfa6092" containerName="payload-processing" containerID="cri-o://f0d41d435ad60f0068740a2a47594205eddec7331ab0334dd8780485364d7d43" gracePeriod=25
Nov 02 02:34:43.030330 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.030220    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg" podUID="79951504-4132-4161-b332-fc7418432ed2" containerName="payload-pre-processing" containerID="cri-o://ce1ff7b5d6a9733200712602a3137ea6d3ca36aae9adb473e810295234711469" gracePeriod=25
Nov 02 02:34:43.184955 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.184889    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz"
Nov 02 02:34:43.282609 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.282582    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b88690-e740-4b6c-9edc-bdd3ddfa6092-kube-api-access-zgwg6\" (UniqueName: \"kubernetes.io/projected/40b88690-e740-4b6c-9edc-bdd3ddfa6092-kube-api-access-zgwg6\") pod \"40b88690-e740-4b6c-9edc-bdd3ddfa6092\" (UID: \"40b88690-e740-4b6c-9edc-bdd3ddfa6092\") "
Nov 02 02:34:43.282762 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.282728    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40b88690-e740-4b6c-9edc-bdd3ddfa6092-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b88690-e740-4b6c-9edc-bdd3ddfa6092-plugins-config-volume\") pod \"40b88690-e740-4b6c-9edc-bdd3ddfa6092\" (UID: \"40b88690-e740-4b6c-9edc-bdd3ddfa6092\") "
Nov 02 02:34:43.283086 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.283060    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b88690-e740-4b6c-9edc-bdd3ddfa6092-plugins-config-volume" pod "40b88690-e740-4b6c-9edc-bdd3ddfa6092" (UID: "40b88690-e740-4b6c-9edc-bdd3ddfa6092"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:34:43.284902 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.284871    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b88690-e740-4b6c-9edc-bdd3ddfa6092-kube-api-access-zgwg6" pod "40b88690-e740-4b6c-9edc-bdd3ddfa6092" (UID: "40b88690-e740-4b6c-9edc-bdd3ddfa6092"). InnerVolumeSpecName "kube-api-access-zgwg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:34:43.290785 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.290763    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg"
Nov 02 02:34:43.383965 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.383928    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79951504-4132-4161-b332-fc7418432ed2-kube-api-access-gt9jp\" (UniqueName: \"kubernetes.io/projected/79951504-4132-4161-b332-fc7418432ed2-kube-api-access-gt9jp\") pod \"79951504-4132-4161-b332-fc7418432ed2\" (UID: \"79951504-4132-4161-b332-fc7418432ed2\") "
Nov 02 02:34:43.384154 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.384096    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79951504-4132-4161-b332-fc7418432ed2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79951504-4132-4161-b332-fc7418432ed2-plugins-config-volume\") pod \"79951504-4132-4161-b332-fc7418432ed2\" (UID: \"79951504-4132-4161-b332-fc7418432ed2\") "
Nov 02 02:34:43.384501 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.384482    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b88690-e740-4b6c-9edc-bdd3ddfa6092-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:34:43.384568 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.384508    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgwg6\" (UniqueName: \"kubernetes.io/projected/40b88690-e740-4b6c-9edc-bdd3ddfa6092-kube-api-access-zgwg6\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:34:43.384568 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.384478    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79951504-4132-4161-b332-fc7418432ed2-plugins-config-volume" pod "79951504-4132-4161-b332-fc7418432ed2" (UID: "79951504-4132-4161-b332-fc7418432ed2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:34:43.386251 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.386231    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79951504-4132-4161-b332-fc7418432ed2-kube-api-access-gt9jp" pod "79951504-4132-4161-b332-fc7418432ed2" (UID: "79951504-4132-4161-b332-fc7418432ed2"). InnerVolumeSpecName "kube-api-access-gt9jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:34:43.485971 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.485886    5639 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79951504-4132-4161-b332-fc7418432ed2-plugins-config-volume\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:34:43.485971 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.485917    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt9jp\" (UniqueName: \"kubernetes.io/projected/79951504-4132-4161-b332-fc7418432ed2-kube-api-access-gt9jp\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:34:43.681667 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.681630    5639 generic.go:356] "Generic (PLEG): container finished" podID="40b88690-e740-4b6c-9edc-bdd3ddfa6092" containerID="f0d41d435ad60f0068740a2a47594205eddec7331ab0334dd8780485364d7d43" exitCode=0
Nov 02 02:34:43.681897 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.681674    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz"
Nov 02 02:34:43.681897 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.681716    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz" event={"ID":"40b88690-e740-4b6c-9edc-bdd3ddfa6092","Type":"ContainerDied","Data":"f0d41d435ad60f0068740a2a47594205eddec7331ab0334dd8780485364d7d43"}
Nov 02 02:34:43.681897 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.681767    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz" event={"ID":"40b88690-e740-4b6c-9edc-bdd3ddfa6092","Type":"ContainerDied","Data":"63f7b1965c61e76d2b880d00d4da12e3d952a2f0ffa44ce2cef1bceea3d3c14c"}
Nov 02 02:34:43.681897 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.681801    5639 scope.go:122] "RemoveContainer" containerID="f0d41d435ad60f0068740a2a47594205eddec7331ab0334dd8780485364d7d43"
Nov 02 02:34:43.683227 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.683200    5639 generic.go:356] "Generic (PLEG): container finished" podID="79951504-4132-4161-b332-fc7418432ed2" containerID="ce1ff7b5d6a9733200712602a3137ea6d3ca36aae9adb473e810295234711469" exitCode=0
Nov 02 02:34:43.683384 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.683238    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg" event={"ID":"79951504-4132-4161-b332-fc7418432ed2","Type":"ContainerDied","Data":"ce1ff7b5d6a9733200712602a3137ea6d3ca36aae9adb473e810295234711469"}
Nov 02 02:34:43.683384 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.683244    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg"
Nov 02 02:34:43.683384 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.683296    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg" event={"ID":"79951504-4132-4161-b332-fc7418432ed2","Type":"ContainerDied","Data":"cdb1485f3b56dadbc104cd7d223dc1b8e1c62499ac61bbe180929ca5c78d679a"}
Nov 02 02:34:43.692051 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.692034    5639 scope.go:122] "RemoveContainer" containerID="f0d41d435ad60f0068740a2a47594205eddec7331ab0334dd8780485364d7d43"
Nov 02 02:34:43.692382 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:34:43.692361    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0d41d435ad60f0068740a2a47594205eddec7331ab0334dd8780485364d7d43\": container with ID starting with f0d41d435ad60f0068740a2a47594205eddec7331ab0334dd8780485364d7d43 not found: ID does not exist" containerID="f0d41d435ad60f0068740a2a47594205eddec7331ab0334dd8780485364d7d43"
Nov 02 02:34:43.692447 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.692399    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d41d435ad60f0068740a2a47594205eddec7331ab0334dd8780485364d7d43"} err="failed to get container status \"f0d41d435ad60f0068740a2a47594205eddec7331ab0334dd8780485364d7d43\": rpc error: code = NotFound desc = could not find container \"f0d41d435ad60f0068740a2a47594205eddec7331ab0334dd8780485364d7d43\": container with ID starting with f0d41d435ad60f0068740a2a47594205eddec7331ab0334dd8780485364d7d43 not found: ID does not exist"
Nov 02 02:34:43.692447 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.692418    5639 scope.go:122] "RemoveContainer" containerID="ce1ff7b5d6a9733200712602a3137ea6d3ca36aae9adb473e810295234711469"
Nov 02 02:34:43.701067 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.701043    5639 scope.go:122] "RemoveContainer" containerID="ce1ff7b5d6a9733200712602a3137ea6d3ca36aae9adb473e810295234711469"
Nov 02 02:34:43.701346 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:34:43.701324    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce1ff7b5d6a9733200712602a3137ea6d3ca36aae9adb473e810295234711469\": container with ID starting with ce1ff7b5d6a9733200712602a3137ea6d3ca36aae9adb473e810295234711469 not found: ID does not exist" containerID="ce1ff7b5d6a9733200712602a3137ea6d3ca36aae9adb473e810295234711469"
Nov 02 02:34:43.701405 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.701354    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce1ff7b5d6a9733200712602a3137ea6d3ca36aae9adb473e810295234711469"} err="failed to get container status \"ce1ff7b5d6a9733200712602a3137ea6d3ca36aae9adb473e810295234711469\": rpc error: code = NotFound desc = could not find container \"ce1ff7b5d6a9733200712602a3137ea6d3ca36aae9adb473e810295234711469\": container with ID starting with ce1ff7b5d6a9733200712602a3137ea6d3ca36aae9adb473e810295234711469 not found: ID does not exist"
Nov 02 02:34:43.845664 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.845623    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz"]
Nov 02 02:34:43.850422 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.850388    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-5f6a0b-7644b7c899-zlhnz"]
Nov 02 02:34:43.957959 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.957917    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg"]
Nov 02 02:34:43.967469 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:43.967439    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-5f6a0b-996645dff-v66fg"]
Nov 02 02:34:45.217138 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:45.217090    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40b88690-e740-4b6c-9edc-bdd3ddfa6092" path="/var/lib/kubelet/pods/40b88690-e740-4b6c-9edc-bdd3ddfa6092/volumes"
Nov 02 02:34:45.217573 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:45.217554    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79951504-4132-4161-b332-fc7418432ed2" path="/var/lib/kubelet/pods/79951504-4132-4161-b332-fc7418432ed2/volumes"
Nov 02 02:34:47.406559 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:47.406526    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh"]
Nov 02 02:34:47.407063 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:47.406762    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh" podUID="6605af90-6fb9-4225-abd8-5af5de1bbfd1" containerName="maas-api" containerID="cri-o://463cc8139a901904ccceb350d75c7467a49498ba98e68ef182ff091aff88fec7" gracePeriod=30
Nov 02 02:34:47.669687 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:47.669604    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh"
Nov 02 02:34:47.711682 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:47.711649    5639 generic.go:356] "Generic (PLEG): container finished" podID="6605af90-6fb9-4225-abd8-5af5de1bbfd1" containerID="463cc8139a901904ccceb350d75c7467a49498ba98e68ef182ff091aff88fec7" exitCode=0
Nov 02 02:34:47.711880 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:47.711699    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh"
Nov 02 02:34:47.711880 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:47.711738    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh" event={"ID":"6605af90-6fb9-4225-abd8-5af5de1bbfd1","Type":"ContainerDied","Data":"463cc8139a901904ccceb350d75c7467a49498ba98e68ef182ff091aff88fec7"}
Nov 02 02:34:47.711880 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:47.711793    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh" event={"ID":"6605af90-6fb9-4225-abd8-5af5de1bbfd1","Type":"ContainerDied","Data":"d2186209e93092d82ba4b70f6a085899e8508a87dd055f9d49fcaa938b8f71c8"}
Nov 02 02:34:47.711880 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:47.711821    5639 scope.go:122] "RemoveContainer" containerID="463cc8139a901904ccceb350d75c7467a49498ba98e68ef182ff091aff88fec7"
Nov 02 02:34:47.721410 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:47.721388    5639 scope.go:122] "RemoveContainer" containerID="463cc8139a901904ccceb350d75c7467a49498ba98e68ef182ff091aff88fec7"
Nov 02 02:34:47.721799 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:34:47.721776    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"463cc8139a901904ccceb350d75c7467a49498ba98e68ef182ff091aff88fec7\": container with ID starting with 463cc8139a901904ccceb350d75c7467a49498ba98e68ef182ff091aff88fec7 not found: ID does not exist" containerID="463cc8139a901904ccceb350d75c7467a49498ba98e68ef182ff091aff88fec7"
Nov 02 02:34:47.721874 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:47.721813    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"463cc8139a901904ccceb350d75c7467a49498ba98e68ef182ff091aff88fec7"} err="failed to get container status \"463cc8139a901904ccceb350d75c7467a49498ba98e68ef182ff091aff88fec7\": rpc error: code = NotFound desc = could not find container \"463cc8139a901904ccceb350d75c7467a49498ba98e68ef182ff091aff88fec7\": container with ID starting with 463cc8139a901904ccceb350d75c7467a49498ba98e68ef182ff091aff88fec7 not found: ID does not exist"
Nov 02 02:34:47.731170 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:47.731145    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6605af90-6fb9-4225-abd8-5af5de1bbfd1-kube-api-access-fgrbm\" (UniqueName: \"kubernetes.io/projected/6605af90-6fb9-4225-abd8-5af5de1bbfd1-kube-api-access-fgrbm\") pod \"6605af90-6fb9-4225-abd8-5af5de1bbfd1\" (UID: \"6605af90-6fb9-4225-abd8-5af5de1bbfd1\") "
Nov 02 02:34:47.731271 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:47.731192    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6605af90-6fb9-4225-abd8-5af5de1bbfd1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6605af90-6fb9-4225-abd8-5af5de1bbfd1-metrics-certs\") pod \"6605af90-6fb9-4225-abd8-5af5de1bbfd1\" (UID: \"6605af90-6fb9-4225-abd8-5af5de1bbfd1\") "
Nov 02 02:34:47.731271 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:47.731213    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6605af90-6fb9-4225-abd8-5af5de1bbfd1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6605af90-6fb9-4225-abd8-5af5de1bbfd1-openshift-service-ca\") pod \"6605af90-6fb9-4225-abd8-5af5de1bbfd1\" (UID: \"6605af90-6fb9-4225-abd8-5af5de1bbfd1\") "
Nov 02 02:34:47.731382 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:47.731326    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6605af90-6fb9-4225-abd8-5af5de1bbfd1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6605af90-6fb9-4225-abd8-5af5de1bbfd1-maas-api-tls\") pod \"6605af90-6fb9-4225-abd8-5af5de1bbfd1\" (UID: \"6605af90-6fb9-4225-abd8-5af5de1bbfd1\") "
Nov 02 02:34:47.731627 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:47.731595    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6605af90-6fb9-4225-abd8-5af5de1bbfd1-openshift-service-ca" pod "6605af90-6fb9-4225-abd8-5af5de1bbfd1" (UID: "6605af90-6fb9-4225-abd8-5af5de1bbfd1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:34:47.731774 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:47.731758    5639 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6605af90-6fb9-4225-abd8-5af5de1bbfd1-openshift-service-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:34:47.733608 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:47.733577    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6605af90-6fb9-4225-abd8-5af5de1bbfd1-kube-api-access-fgrbm" pod "6605af90-6fb9-4225-abd8-5af5de1bbfd1" (UID: "6605af90-6fb9-4225-abd8-5af5de1bbfd1"). InnerVolumeSpecName "kube-api-access-fgrbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:34:47.733714 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:47.733642    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6605af90-6fb9-4225-abd8-5af5de1bbfd1-metrics-certs" pod "6605af90-6fb9-4225-abd8-5af5de1bbfd1" (UID: "6605af90-6fb9-4225-abd8-5af5de1bbfd1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:34:47.733714 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:47.733652    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6605af90-6fb9-4225-abd8-5af5de1bbfd1-maas-api-tls" pod "6605af90-6fb9-4225-abd8-5af5de1bbfd1" (UID: "6605af90-6fb9-4225-abd8-5af5de1bbfd1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:34:47.833124 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:47.833081    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgrbm\" (UniqueName: \"kubernetes.io/projected/6605af90-6fb9-4225-abd8-5af5de1bbfd1-kube-api-access-fgrbm\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:34:47.833124 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:47.833126    5639 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6605af90-6fb9-4225-abd8-5af5de1bbfd1-metrics-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:34:47.833409 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:47.833143    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6605af90-6fb9-4225-abd8-5af5de1bbfd1-maas-api-tls\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:34:48.131701 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:48.131658    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh"]
Nov 02 02:34:48.135982 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:48.135952    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5f6a0b-d9fd97584-bdjsh"]
Nov 02 02:34:49.218535 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:49.218499    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6605af90-6fb9-4225-abd8-5af5de1bbfd1" path="/var/lib/kubelet/pods/6605af90-6fb9-4225-abd8-5af5de1bbfd1/volumes"
Nov 02 02:34:49.625941 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:49.625903    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb"]
Nov 02 02:34:49.675361 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:49.675317    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5f6a0b-hb8cb"]
Nov 02 02:34:51.218309 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:51.218250    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82560d25-c97a-4bff-96cd-943f7746489f" path="/var/lib/kubelet/pods/82560d25-c97a-4bff-96cd-943f7746489f/volumes"
Nov 02 02:34:52.500784 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:52.500734    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:34:56.290609 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:56.290576    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"]
Nov 02 02:34:56.291102 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:56.290781    5639 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4" podUID="dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6" containerName="istio-proxy" containerID="cri-o://5b7c02f9d5fb095af75ec3b6580f15ba060b312be9c5babcf4ab353198757b4c" gracePeriod=30
Nov 02 02:34:56.635883 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:56.635835    5639 patch_prober.go:29] interesting pod/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:34:56.636102 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:34:56.635904    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4" podUID="dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:34:57.420742 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:34:57.420703    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:35:01.552043 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.552009    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:35:01.679910 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.679874    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-envoy\") pod \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") "
Nov 02 02:35:01.680127 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.679937    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-podinfo\") pod \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") "
Nov 02 02:35:01.680127 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.679976    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-kube-api-access-d4cqb\" (UniqueName: \"kubernetes.io/projected/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-kube-api-access-d4cqb\") pod \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") "
Nov 02 02:35:01.680127 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.680015    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-credential-socket\") pod \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") "
Nov 02 02:35:01.680127 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.680041    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-workload-socket\") pod \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") "
Nov 02 02:35:01.680127 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.680114    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-token\" (UniqueName: \"kubernetes.io/projected/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-token\") pod \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") "
Nov 02 02:35:01.680469 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.680142    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-data\") pod \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") "
Nov 02 02:35:01.680469 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.680176    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-workload-certs\") pod \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") "
Nov 02 02:35:01.680469 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.680254    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istiod-ca-cert\") pod \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\" (UID: \"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6\") "
Nov 02 02:35:01.680469 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.680359    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-workload-socket" pod "dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6" (UID: "dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:35:01.680469 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.680376    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-credential-socket" pod "dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6" (UID: "dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:35:01.680762 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.680579    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-data" pod "dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6" (UID: "dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:35:01.680762 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.680746    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-workload-certs" pod "dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6" (UID: "dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:35:01.680846 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.680761    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istiod-ca-cert" pod "dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6" (UID: "dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:35:01.680945 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.680877    5639 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-credential-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:01.680945 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.680924    5639 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-workload-socket\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:01.680945 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.680940    5639 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-data\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:01.682384 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.682358    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-kube-api-access-d4cqb" pod "dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6" (UID: "dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6"). InnerVolumeSpecName "kube-api-access-d4cqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:35:01.682671 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.682650    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-token" pod "dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6" (UID: "dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:35:01.682767 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.682706    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-envoy" pod "dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6" (UID: "dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:35:01.682827 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.682780    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-podinfo" pod "dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6" (UID: "dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:35:01.782563 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.782526    5639 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-podinfo\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:01.782563 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.782557    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4cqb\" (UniqueName: \"kubernetes.io/projected/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-kube-api-access-d4cqb\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:01.782563 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.782567    5639 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-token\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:01.782814 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.782579    5639 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-workload-certs\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:01.782814 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.782592    5639 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istiod-ca-cert\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:01.782814 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.782600    5639 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6-istio-envoy\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:01.788746 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.788712    5639 generic.go:356] "Generic (PLEG): container finished" podID="dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6" containerID="5b7c02f9d5fb095af75ec3b6580f15ba060b312be9c5babcf4ab353198757b4c" exitCode=0
Nov 02 02:35:01.788909 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.788765    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"
Nov 02 02:35:01.788909 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.788771    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4" event={"ID":"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6","Type":"ContainerDied","Data":"5b7c02f9d5fb095af75ec3b6580f15ba060b312be9c5babcf4ab353198757b4c"}
Nov 02 02:35:01.788909 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.788824    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4" event={"ID":"dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6","Type":"ContainerDied","Data":"e7a7a0eb12685c9addf0c14d49ed09fd2e33a72e86db41841e08705ab0a06c20"}
Nov 02 02:35:01.788909 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.788854    5639 scope.go:122] "RemoveContainer" containerID="5b7c02f9d5fb095af75ec3b6580f15ba060b312be9c5babcf4ab353198757b4c"
Nov 02 02:35:01.796850 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.796824    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:35:01.799524 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.799505    5639 scope.go:122] "RemoveContainer" containerID="22599a1fcc32b808a6f42b765acbed80f8b8a48ebc874e5850e7de44d4d8bd55"
Nov 02 02:35:01.809614 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.809592    5639 scope.go:122] "RemoveContainer" containerID="5b7c02f9d5fb095af75ec3b6580f15ba060b312be9c5babcf4ab353198757b4c"
Nov 02 02:35:01.809911 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:35:01.809889    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b7c02f9d5fb095af75ec3b6580f15ba060b312be9c5babcf4ab353198757b4c\": container with ID starting with 5b7c02f9d5fb095af75ec3b6580f15ba060b312be9c5babcf4ab353198757b4c not found: ID does not exist" containerID="5b7c02f9d5fb095af75ec3b6580f15ba060b312be9c5babcf4ab353198757b4c"
Nov 02 02:35:01.810001 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.809925    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b7c02f9d5fb095af75ec3b6580f15ba060b312be9c5babcf4ab353198757b4c"} err="failed to get container status \"5b7c02f9d5fb095af75ec3b6580f15ba060b312be9c5babcf4ab353198757b4c\": rpc error: code = NotFound desc = could not find container \"5b7c02f9d5fb095af75ec3b6580f15ba060b312be9c5babcf4ab353198757b4c\": container with ID starting with 5b7c02f9d5fb095af75ec3b6580f15ba060b312be9c5babcf4ab353198757b4c not found: ID does not exist"
Nov 02 02:35:01.810001 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.809954    5639 scope.go:122] "RemoveContainer" containerID="22599a1fcc32b808a6f42b765acbed80f8b8a48ebc874e5850e7de44d4d8bd55"
Nov 02 02:35:01.810247 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:35:01.810230    5639 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22599a1fcc32b808a6f42b765acbed80f8b8a48ebc874e5850e7de44d4d8bd55\": container with ID starting with 22599a1fcc32b808a6f42b765acbed80f8b8a48ebc874e5850e7de44d4d8bd55 not found: ID does not exist" containerID="22599a1fcc32b808a6f42b765acbed80f8b8a48ebc874e5850e7de44d4d8bd55"
Nov 02 02:35:01.810321 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.810255    5639 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22599a1fcc32b808a6f42b765acbed80f8b8a48ebc874e5850e7de44d4d8bd55"} err="failed to get container status \"22599a1fcc32b808a6f42b765acbed80f8b8a48ebc874e5850e7de44d4d8bd55\": rpc error: code = NotFound desc = could not find container \"22599a1fcc32b808a6f42b765acbed80f8b8a48ebc874e5850e7de44d4d8bd55\": container with ID starting with 22599a1fcc32b808a6f42b765acbed80f8b8a48ebc874e5850e7de44d4d8bd55 not found: ID does not exist"
Nov 02 02:35:01.956841 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.956750    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"]
Nov 02 02:35:01.960741 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:01.960712    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-5f6a0b-openshift-default-77889658c7-znkg4"]
Nov 02 02:35:03.237885 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:03.236008    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6" path="/var/lib/kubelet/pods/dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6/volumes"
Nov 02 02:35:04.102509 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:04.102465    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:35:04.102905 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:04.102872    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qnpm7" podUID="70ec0d83-9e6c-4e20-82b6-2f09909fc85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:35:04.258537 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:04.258482    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:35:04.259176 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:04.259126    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q6wpx" podUID="db2e4537-de21-48f8-858e-1ead87636161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:35:12.208403 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:12.208364    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:35:20.429304 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.429250    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w6tql"]
Nov 02 02:35:20.429701 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.429687    5639 cpu_manager.go:414] "containerMap: removing container" podUID="40b88690-e740-4b6c-9edc-bdd3ddfa6092" containerName="payload-processing"
Nov 02 02:35:20.429749 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.429700    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b88690-e740-4b6c-9edc-bdd3ddfa6092" containerName="payload-processing"
Nov 02 02:35:20.429749 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.429716    5639 cpu_manager.go:414] "containerMap: removing container" podUID="6605af90-6fb9-4225-abd8-5af5de1bbfd1" containerName="maas-api"
Nov 02 02:35:20.429749 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.429721    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6605af90-6fb9-4225-abd8-5af5de1bbfd1" containerName="maas-api"
Nov 02 02:35:20.429749 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.429738    5639 cpu_manager.go:414] "containerMap: removing container" podUID="79951504-4132-4161-b332-fc7418432ed2" containerName="payload-pre-processing"
Nov 02 02:35:20.429749 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.429744    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79951504-4132-4161-b332-fc7418432ed2" containerName="payload-pre-processing"
Nov 02 02:35:20.429914 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.429760    5639 cpu_manager.go:414] "containerMap: removing container" podUID="82560d25-c97a-4bff-96cd-943f7746489f" containerName="revoke-keys"
Nov 02 02:35:20.429914 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.429765    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82560d25-c97a-4bff-96cd-943f7746489f" containerName="revoke-keys"
Nov 02 02:35:20.429914 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.429773    5639 cpu_manager.go:414] "containerMap: removing container" podUID="dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6" containerName="istio-proxy"
Nov 02 02:35:20.429914 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.429777    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6" containerName="istio-proxy"
Nov 02 02:35:20.429914 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.429787    5639 cpu_manager.go:414] "containerMap: removing container" podUID="dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6" containerName="istio-proxy"
Nov 02 02:35:20.429914 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.429792    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6" containerName="istio-proxy"
Nov 02 02:35:20.429914 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.429854    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="6605af90-6fb9-4225-abd8-5af5de1bbfd1" containerName="maas-api"
Nov 02 02:35:20.429914 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.429864    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="79951504-4132-4161-b332-fc7418432ed2" containerName="payload-pre-processing"
Nov 02 02:35:20.429914 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.429873    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="82560d25-c97a-4bff-96cd-943f7746489f" containerName="revoke-keys"
Nov 02 02:35:20.429914 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.429880    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6" containerName="istio-proxy"
Nov 02 02:35:20.429914 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.429888    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="dff6e2ea-c87e-411d-b3a9-1a17fc12d8c6" containerName="istio-proxy"
Nov 02 02:35:20.429914 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.429894    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b88690-e740-4b6c-9edc-bdd3ddfa6092" containerName="payload-processing"
Nov 02 02:35:20.433120 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.433103    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w6tql"
Nov 02 02:35:20.490671 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.490631    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w6tql"]
Nov 02 02:35:20.537699 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.537669    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:35:20.537699 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.537692    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:35:20.537917 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.537771    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:35:20.537917 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.537806    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhr9x\""
Nov 02 02:35:20.564720 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.564694    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb81ec93-3689-4fc9-8ce0-8c17acbe88cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-w6tql\" (UID: \"fb81ec93-3689-4fc9-8ce0-8c17acbe88cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w6tql"
Nov 02 02:35:20.564861 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.564730    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs579\" (UniqueName: \"kubernetes.io/projected/fb81ec93-3689-4fc9-8ce0-8c17acbe88cc-kube-api-access-fs579\") pod \"maas-api-revoke-sub-f03629db-w6tql\" (UID: \"fb81ec93-3689-4fc9-8ce0-8c17acbe88cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w6tql"
Nov 02 02:35:20.564861 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.564837    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb81ec93-3689-4fc9-8ce0-8c17acbe88cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-w6tql\" (UID: \"fb81ec93-3689-4fc9-8ce0-8c17acbe88cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w6tql"
Nov 02 02:35:20.665681 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.665641    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb81ec93-3689-4fc9-8ce0-8c17acbe88cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-w6tql\" (UID: \"fb81ec93-3689-4fc9-8ce0-8c17acbe88cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w6tql"
Nov 02 02:35:20.665861 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.665739    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb81ec93-3689-4fc9-8ce0-8c17acbe88cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-w6tql\" (UID: \"fb81ec93-3689-4fc9-8ce0-8c17acbe88cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w6tql"
Nov 02 02:35:20.665861 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.665769    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs579\" (UniqueName: \"kubernetes.io/projected/fb81ec93-3689-4fc9-8ce0-8c17acbe88cc-kube-api-access-fs579\") pod \"maas-api-revoke-sub-f03629db-w6tql\" (UID: \"fb81ec93-3689-4fc9-8ce0-8c17acbe88cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w6tql"
Nov 02 02:35:20.666321 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.666267    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb81ec93-3689-4fc9-8ce0-8c17acbe88cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-w6tql\" (UID: \"fb81ec93-3689-4fc9-8ce0-8c17acbe88cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w6tql"
Nov 02 02:35:20.668238 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.668216    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb81ec93-3689-4fc9-8ce0-8c17acbe88cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-w6tql\" (UID: \"fb81ec93-3689-4fc9-8ce0-8c17acbe88cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w6tql"
Nov 02 02:35:20.726558 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.726487    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs579\" (UniqueName: \"kubernetes.io/projected/fb81ec93-3689-4fc9-8ce0-8c17acbe88cc-kube-api-access-fs579\") pod \"maas-api-revoke-sub-f03629db-w6tql\" (UID: \"fb81ec93-3689-4fc9-8ce0-8c17acbe88cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w6tql"
Nov 02 02:35:20.745842 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:20.745812    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w6tql"
Nov 02 02:35:21.120050 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:21.119939    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w6tql"]
Nov 02 02:35:21.122746 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:35:21.122711    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb81ec93_3689_4fc9_8ce0_8c17acbe88cc.slice/crio-2a3fd285cc236ef87bdc0ff8da8c39a4586ffe68b86562991f5f5d72cab39fb7 WatchSource:0}: Error finding container 2a3fd285cc236ef87bdc0ff8da8c39a4586ffe68b86562991f5f5d72cab39fb7: Status 404 returned error can't find the container with id 2a3fd285cc236ef87bdc0ff8da8c39a4586ffe68b86562991f5f5d72cab39fb7
Nov 02 02:35:21.992985 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:21.992958    5639 generic.go:356] "Generic (PLEG): container finished" podID="fb81ec93-3689-4fc9-8ce0-8c17acbe88cc" containerID="31e6b256d9f9365a5f20fb9a06d1c87ae7f0a4604f2ec914c4d140631c7f63b5" exitCode=0
Nov 02 02:35:21.993363 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:21.993047    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w6tql" event={"ID":"fb81ec93-3689-4fc9-8ce0-8c17acbe88cc","Type":"ContainerDied","Data":"31e6b256d9f9365a5f20fb9a06d1c87ae7f0a4604f2ec914c4d140631c7f63b5"}
Nov 02 02:35:21.993363 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:21.993098    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w6tql" event={"ID":"fb81ec93-3689-4fc9-8ce0-8c17acbe88cc","Type":"ContainerStarted","Data":"2a3fd285cc236ef87bdc0ff8da8c39a4586ffe68b86562991f5f5d72cab39fb7"}
Nov 02 02:35:23.135415 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:23.135383    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w6tql"
Nov 02 02:35:23.288958 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:23.288868    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb81ec93-3689-4fc9-8ce0-8c17acbe88cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb81ec93-3689-4fc9-8ce0-8c17acbe88cc-maas-api-openshift-ca\") pod \"fb81ec93-3689-4fc9-8ce0-8c17acbe88cc\" (UID: \"fb81ec93-3689-4fc9-8ce0-8c17acbe88cc\") "
Nov 02 02:35:23.289117 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:23.289024    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb81ec93-3689-4fc9-8ce0-8c17acbe88cc-kube-api-access-fs579\" (UniqueName: \"kubernetes.io/projected/fb81ec93-3689-4fc9-8ce0-8c17acbe88cc-kube-api-access-fs579\") pod \"fb81ec93-3689-4fc9-8ce0-8c17acbe88cc\" (UID: \"fb81ec93-3689-4fc9-8ce0-8c17acbe88cc\") "
Nov 02 02:35:23.289168 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:23.289111    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb81ec93-3689-4fc9-8ce0-8c17acbe88cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb81ec93-3689-4fc9-8ce0-8c17acbe88cc-maas-api-xks-ca\") pod \"fb81ec93-3689-4fc9-8ce0-8c17acbe88cc\" (UID: \"fb81ec93-3689-4fc9-8ce0-8c17acbe88cc\") "
Nov 02 02:35:23.289251 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:23.289230    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb81ec93-3689-4fc9-8ce0-8c17acbe88cc-maas-api-openshift-ca" pod "fb81ec93-3689-4fc9-8ce0-8c17acbe88cc" (UID: "fb81ec93-3689-4fc9-8ce0-8c17acbe88cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:35:23.289459 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:23.289441    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb81ec93-3689-4fc9-8ce0-8c17acbe88cc-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:23.291373 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:23.291348    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb81ec93-3689-4fc9-8ce0-8c17acbe88cc-kube-api-access-fs579" pod "fb81ec93-3689-4fc9-8ce0-8c17acbe88cc" (UID: "fb81ec93-3689-4fc9-8ce0-8c17acbe88cc"). InnerVolumeSpecName "kube-api-access-fs579". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:35:23.291477 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:23.291407    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb81ec93-3689-4fc9-8ce0-8c17acbe88cc-maas-api-xks-ca" pod "fb81ec93-3689-4fc9-8ce0-8c17acbe88cc" (UID: "fb81ec93-3689-4fc9-8ce0-8c17acbe88cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:35:23.390806 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:23.390764    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs579\" (UniqueName: \"kubernetes.io/projected/fb81ec93-3689-4fc9-8ce0-8c17acbe88cc-kube-api-access-fs579\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:23.390806 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:23.390801    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb81ec93-3689-4fc9-8ce0-8c17acbe88cc-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:24.003884 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:24.003856    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w6tql"
Nov 02 02:35:24.004071 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:24.003856    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w6tql" event={"ID":"fb81ec93-3689-4fc9-8ce0-8c17acbe88cc","Type":"ContainerDied","Data":"2a3fd285cc236ef87bdc0ff8da8c39a4586ffe68b86562991f5f5d72cab39fb7"}
Nov 02 02:35:24.004071 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:24.004005    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a3fd285cc236ef87bdc0ff8da8c39a4586ffe68b86562991f5f5d72cab39fb7"
Nov 02 02:35:26.056163 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:26.056128    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5rtbc"]
Nov 02 02:35:26.106383 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:26.106353    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5rtbc"]
Nov 02 02:35:27.216617 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:27.216576    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb6129b9-e5b7-412a-a3f6-d7e8694db3f7" path="/var/lib/kubelet/pods/cb6129b9-e5b7-412a-a3f6-d7e8694db3f7/volumes"
Nov 02 02:35:27.420385 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:35:27.420345    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:35:33.209310 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:33.209261    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:35:40.381305 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:40.379935    5639 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fd65x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:35:40.381305 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:40.380222    5639 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fd65x" podUID="faf2916d-803b-4f14-a2b1-1c45b742fc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:35:41.373348 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:41.373309    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gqgr9"]
Nov 02 02:35:41.373739 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:41.373725    5639 cpu_manager.go:414] "containerMap: removing container" podUID="fb81ec93-3689-4fc9-8ce0-8c17acbe88cc" containerName="revoke-keys"
Nov 02 02:35:41.373786 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:41.373741    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb81ec93-3689-4fc9-8ce0-8c17acbe88cc" containerName="revoke-keys"
Nov 02 02:35:41.373847 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:41.373838    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb81ec93-3689-4fc9-8ce0-8c17acbe88cc" containerName="revoke-keys"
Nov 02 02:35:41.378400 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:41.378383    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gqgr9"
Nov 02 02:35:41.434515 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:41.434478    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gqgr9"]
Nov 02 02:35:41.443021 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:41.442991    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64171361-a2ca-468a-b389-26fb4a438fff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-gqgr9\" (UID: \"64171361-a2ca-468a-b389-26fb4a438fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gqgr9"
Nov 02 02:35:41.443209 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:41.443107    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksrcr\" (UniqueName: \"kubernetes.io/projected/64171361-a2ca-468a-b389-26fb4a438fff-kube-api-access-ksrcr\") pod \"maas-api-revoke-sub-0165d7d2-gqgr9\" (UID: \"64171361-a2ca-468a-b389-26fb4a438fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gqgr9"
Nov 02 02:35:41.443209 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:41.443133    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64171361-a2ca-468a-b389-26fb4a438fff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-gqgr9\" (UID: \"64171361-a2ca-468a-b389-26fb4a438fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gqgr9"
Nov 02 02:35:41.481904 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:41.481870    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:35:41.481904 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:41.481875    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:35:41.482134 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:41.481929    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:35:41.482134 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:41.481941    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhr9x\""
Nov 02 02:35:41.543938 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:41.543891    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksrcr\" (UniqueName: \"kubernetes.io/projected/64171361-a2ca-468a-b389-26fb4a438fff-kube-api-access-ksrcr\") pod \"maas-api-revoke-sub-0165d7d2-gqgr9\" (UID: \"64171361-a2ca-468a-b389-26fb4a438fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gqgr9"
Nov 02 02:35:41.544162 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:41.543945    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64171361-a2ca-468a-b389-26fb4a438fff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-gqgr9\" (UID: \"64171361-a2ca-468a-b389-26fb4a438fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gqgr9"
Nov 02 02:35:41.544162 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:41.544078    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64171361-a2ca-468a-b389-26fb4a438fff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-gqgr9\" (UID: \"64171361-a2ca-468a-b389-26fb4a438fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gqgr9"
Nov 02 02:35:41.544909 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:41.544884    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64171361-a2ca-468a-b389-26fb4a438fff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-gqgr9\" (UID: \"64171361-a2ca-468a-b389-26fb4a438fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gqgr9"
Nov 02 02:35:41.546591 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:41.546571    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64171361-a2ca-468a-b389-26fb4a438fff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-gqgr9\" (UID: \"64171361-a2ca-468a-b389-26fb4a438fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gqgr9"
Nov 02 02:35:41.604914 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:41.604883    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksrcr\" (UniqueName: \"kubernetes.io/projected/64171361-a2ca-468a-b389-26fb4a438fff-kube-api-access-ksrcr\") pod \"maas-api-revoke-sub-0165d7d2-gqgr9\" (UID: \"64171361-a2ca-468a-b389-26fb4a438fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gqgr9"
Nov 02 02:35:41.691570 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:41.691474    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gqgr9"
Nov 02 02:35:42.065939 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:42.065915    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gqgr9"]
Nov 02 02:35:42.068543 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:35:42.068511    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64171361_a2ca_468a_b389_26fb4a438fff.slice/crio-2bdbe1baf293e01100c2bcb88266bc29134e94062ed463cd2efe6aeddfc1fedb WatchSource:0}: Error finding container 2bdbe1baf293e01100c2bcb88266bc29134e94062ed463cd2efe6aeddfc1fedb: Status 404 returned error can't find the container with id 2bdbe1baf293e01100c2bcb88266bc29134e94062ed463cd2efe6aeddfc1fedb
Nov 02 02:35:42.107663 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:42.107631    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gqgr9" event={"ID":"64171361-a2ca-468a-b389-26fb4a438fff","Type":"ContainerStarted","Data":"2bdbe1baf293e01100c2bcb88266bc29134e94062ed463cd2efe6aeddfc1fedb"}
Nov 02 02:35:43.114931 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:43.114885    5639 generic.go:356] "Generic (PLEG): container finished" podID="64171361-a2ca-468a-b389-26fb4a438fff" containerID="1c79013c20255c59a0c8b9ab19631a8111c3f6892e234335ca891ea0d239b10c" exitCode=0
Nov 02 02:35:43.115524 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:43.114959    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gqgr9" event={"ID":"64171361-a2ca-468a-b389-26fb4a438fff","Type":"ContainerDied","Data":"1c79013c20255c59a0c8b9ab19631a8111c3f6892e234335ca891ea0d239b10c"}
Nov 02 02:35:44.259680 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:44.259633    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gqgr9"
Nov 02 02:35:44.274079 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:44.274045    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64171361-a2ca-468a-b389-26fb4a438fff-kube-api-access-ksrcr\" (UniqueName: \"kubernetes.io/projected/64171361-a2ca-468a-b389-26fb4a438fff-kube-api-access-ksrcr\") pod \"64171361-a2ca-468a-b389-26fb4a438fff\" (UID: \"64171361-a2ca-468a-b389-26fb4a438fff\") "
Nov 02 02:35:44.274264 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:44.274139    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64171361-a2ca-468a-b389-26fb4a438fff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64171361-a2ca-468a-b389-26fb4a438fff-maas-api-xks-ca\") pod \"64171361-a2ca-468a-b389-26fb4a438fff\" (UID: \"64171361-a2ca-468a-b389-26fb4a438fff\") "
Nov 02 02:35:44.274264 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:44.274233    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64171361-a2ca-468a-b389-26fb4a438fff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64171361-a2ca-468a-b389-26fb4a438fff-maas-api-openshift-ca\") pod \"64171361-a2ca-468a-b389-26fb4a438fff\" (UID: \"64171361-a2ca-468a-b389-26fb4a438fff\") "
Nov 02 02:35:44.274630 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:44.274599    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64171361-a2ca-468a-b389-26fb4a438fff-maas-api-openshift-ca" pod "64171361-a2ca-468a-b389-26fb4a438fff" (UID: "64171361-a2ca-468a-b389-26fb4a438fff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:35:44.276415 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:44.276388    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64171361-a2ca-468a-b389-26fb4a438fff-maas-api-xks-ca" pod "64171361-a2ca-468a-b389-26fb4a438fff" (UID: "64171361-a2ca-468a-b389-26fb4a438fff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:35:44.276518 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:44.276447    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64171361-a2ca-468a-b389-26fb4a438fff-kube-api-access-ksrcr" pod "64171361-a2ca-468a-b389-26fb4a438fff" (UID: "64171361-a2ca-468a-b389-26fb4a438fff"). InnerVolumeSpecName "kube-api-access-ksrcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:35:44.375488 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:44.375390    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksrcr\" (UniqueName: \"kubernetes.io/projected/64171361-a2ca-468a-b389-26fb4a438fff-kube-api-access-ksrcr\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:44.375488 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:44.375428    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64171361-a2ca-468a-b389-26fb4a438fff-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:44.375488 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:44.375442    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64171361-a2ca-468a-b389-26fb4a438fff-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:45.126004 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:45.125968    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gqgr9" event={"ID":"64171361-a2ca-468a-b389-26fb4a438fff","Type":"ContainerDied","Data":"2bdbe1baf293e01100c2bcb88266bc29134e94062ed463cd2efe6aeddfc1fedb"}
Nov 02 02:35:45.126213 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:45.126050    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bdbe1baf293e01100c2bcb88266bc29134e94062ed463cd2efe6aeddfc1fedb"
Nov 02 02:35:45.126213 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:45.125985    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gqgr9"
Nov 02 02:35:51.045796 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:51.045761    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-dn28l"]
Nov 02 02:35:51.067195 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:51.067156    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-dn28l"]
Nov 02 02:35:51.224129 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:51.224081    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aa7252e-e373-47b5-8664-0931881b390a" path="/var/lib/kubelet/pods/5aa7252e-e373-47b5-8664-0931881b390a/volumes"
Nov 02 02:35:53.156266 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:53.156231    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-x4wrb"]
Nov 02 02:35:53.156693 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:53.156671    5639 cpu_manager.go:414] "containerMap: removing container" podUID="64171361-a2ca-468a-b389-26fb4a438fff" containerName="revoke-keys"
Nov 02 02:35:53.156693 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:53.156684    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64171361-a2ca-468a-b389-26fb4a438fff" containerName="revoke-keys"
Nov 02 02:35:53.156791 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:53.156780    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="64171361-a2ca-468a-b389-26fb4a438fff" containerName="revoke-keys"
Nov 02 02:35:53.160404 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:53.160383    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-x4wrb"
Nov 02 02:35:53.218104 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:53.218069    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-x4wrb"]
Nov 02 02:35:53.264921 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:53.264881    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:35:53.265162 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:53.265141    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:35:53.265236 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:53.265222    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhr9x\""
Nov 02 02:35:53.265319 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:53.265246    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:35:53.270638 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:53.270614    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzl9m\" (UniqueName: \"kubernetes.io/projected/645900e9-c3f9-43f6-bd01-cb83dc083995-kube-api-access-wzl9m\") pod \"maas-api-revoke-sub-6b19338f-x4wrb\" (UID: \"645900e9-c3f9-43f6-bd01-cb83dc083995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-x4wrb"
Nov 02 02:35:53.270756 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:53.270665    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/645900e9-c3f9-43f6-bd01-cb83dc083995-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-x4wrb\" (UID: \"645900e9-c3f9-43f6-bd01-cb83dc083995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-x4wrb"
Nov 02 02:35:53.270808 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:53.270791    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/645900e9-c3f9-43f6-bd01-cb83dc083995-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-x4wrb\" (UID: \"645900e9-c3f9-43f6-bd01-cb83dc083995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-x4wrb"
Nov 02 02:35:53.371929 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:53.371888    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzl9m\" (UniqueName: \"kubernetes.io/projected/645900e9-c3f9-43f6-bd01-cb83dc083995-kube-api-access-wzl9m\") pod \"maas-api-revoke-sub-6b19338f-x4wrb\" (UID: \"645900e9-c3f9-43f6-bd01-cb83dc083995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-x4wrb"
Nov 02 02:35:53.372134 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:53.371947    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/645900e9-c3f9-43f6-bd01-cb83dc083995-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-x4wrb\" (UID: \"645900e9-c3f9-43f6-bd01-cb83dc083995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-x4wrb"
Nov 02 02:35:53.372134 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:53.371979    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/645900e9-c3f9-43f6-bd01-cb83dc083995-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-x4wrb\" (UID: \"645900e9-c3f9-43f6-bd01-cb83dc083995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-x4wrb"
Nov 02 02:35:53.372677 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:53.372643    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/645900e9-c3f9-43f6-bd01-cb83dc083995-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-x4wrb\" (UID: \"645900e9-c3f9-43f6-bd01-cb83dc083995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-x4wrb"
Nov 02 02:35:53.374554 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:53.374530    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/645900e9-c3f9-43f6-bd01-cb83dc083995-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-x4wrb\" (UID: \"645900e9-c3f9-43f6-bd01-cb83dc083995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-x4wrb"
Nov 02 02:35:53.433064 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:53.432987    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzl9m\" (UniqueName: \"kubernetes.io/projected/645900e9-c3f9-43f6-bd01-cb83dc083995-kube-api-access-wzl9m\") pod \"maas-api-revoke-sub-6b19338f-x4wrb\" (UID: \"645900e9-c3f9-43f6-bd01-cb83dc083995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-x4wrb"
Nov 02 02:35:53.473956 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:53.473916    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-x4wrb"
Nov 02 02:35:53.847931 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:53.847899    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-x4wrb"]
Nov 02 02:35:53.850793 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:35:53.850767    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645900e9_c3f9_43f6_bd01_cb83dc083995.slice/crio-be5df86687a6a4ac3b913d5d973b88695a21abf01a290444eb39b0ae752d1d33 WatchSource:0}: Error finding container be5df86687a6a4ac3b913d5d973b88695a21abf01a290444eb39b0ae752d1d33: Status 404 returned error can't find the container with id be5df86687a6a4ac3b913d5d973b88695a21abf01a290444eb39b0ae752d1d33
Nov 02 02:35:54.175349 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:54.175312    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-x4wrb" event={"ID":"645900e9-c3f9-43f6-bd01-cb83dc083995","Type":"ContainerStarted","Data":"be5df86687a6a4ac3b913d5d973b88695a21abf01a290444eb39b0ae752d1d33"}
Nov 02 02:35:55.182249 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:55.182208    5639 generic.go:356] "Generic (PLEG): container finished" podID="645900e9-c3f9-43f6-bd01-cb83dc083995" containerID="aa4a56222bff2548de2fb0a60e756a78c3dce8acc497872b8587d0755939c966" exitCode=0
Nov 02 02:35:55.182702 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:55.182313    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-x4wrb" event={"ID":"645900e9-c3f9-43f6-bd01-cb83dc083995","Type":"ContainerDied","Data":"aa4a56222bff2548de2fb0a60e756a78c3dce8acc497872b8587d0755939c966"}
Nov 02 02:35:56.327789 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:56.327763    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-x4wrb"
Nov 02 02:35:56.404982 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:56.404945    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/645900e9-c3f9-43f6-bd01-cb83dc083995-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/645900e9-c3f9-43f6-bd01-cb83dc083995-maas-api-openshift-ca\") pod \"645900e9-c3f9-43f6-bd01-cb83dc083995\" (UID: \"645900e9-c3f9-43f6-bd01-cb83dc083995\") "
Nov 02 02:35:56.405205 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:56.405013    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/645900e9-c3f9-43f6-bd01-cb83dc083995-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/645900e9-c3f9-43f6-bd01-cb83dc083995-maas-api-xks-ca\") pod \"645900e9-c3f9-43f6-bd01-cb83dc083995\" (UID: \"645900e9-c3f9-43f6-bd01-cb83dc083995\") "
Nov 02 02:35:56.405205 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:56.405162    5639 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/645900e9-c3f9-43f6-bd01-cb83dc083995-kube-api-access-wzl9m\" (UniqueName: \"kubernetes.io/projected/645900e9-c3f9-43f6-bd01-cb83dc083995-kube-api-access-wzl9m\") pod \"645900e9-c3f9-43f6-bd01-cb83dc083995\" (UID: \"645900e9-c3f9-43f6-bd01-cb83dc083995\") "
Nov 02 02:35:56.405455 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:56.405428    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/645900e9-c3f9-43f6-bd01-cb83dc083995-maas-api-openshift-ca" pod "645900e9-c3f9-43f6-bd01-cb83dc083995" (UID: "645900e9-c3f9-43f6-bd01-cb83dc083995"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:35:56.405579 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:56.405561    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/645900e9-c3f9-43f6-bd01-cb83dc083995-maas-api-openshift-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:56.407350 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:56.407321    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/645900e9-c3f9-43f6-bd01-cb83dc083995-maas-api-xks-ca" pod "645900e9-c3f9-43f6-bd01-cb83dc083995" (UID: "645900e9-c3f9-43f6-bd01-cb83dc083995"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:35:56.407454 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:56.407403    5639 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645900e9-c3f9-43f6-bd01-cb83dc083995-kube-api-access-wzl9m" pod "645900e9-c3f9-43f6-bd01-cb83dc083995" (UID: "645900e9-c3f9-43f6-bd01-cb83dc083995"). InnerVolumeSpecName "kube-api-access-wzl9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:35:56.507044 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:56.506953    5639 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/645900e9-c3f9-43f6-bd01-cb83dc083995-maas-api-xks-ca\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:56.507044 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:56.506989    5639 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzl9m\" (UniqueName: \"kubernetes.io/projected/645900e9-c3f9-43f6-bd01-cb83dc083995-kube-api-access-wzl9m\") on node \"ip-10-0-136-53.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:57.193797 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:57.193761    5639 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-x4wrb"
Nov 02 02:35:57.193993 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:57.193761    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-x4wrb" event={"ID":"645900e9-c3f9-43f6-bd01-cb83dc083995","Type":"ContainerDied","Data":"be5df86687a6a4ac3b913d5d973b88695a21abf01a290444eb39b0ae752d1d33"}
Nov 02 02:35:57.193993 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:35:57.193914    5639 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be5df86687a6a4ac3b913d5d973b88695a21abf01a290444eb39b0ae752d1d33"
Nov 02 02:35:57.421543 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:35:57.421503    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:36:05.393643 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:05.393608    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-595dbbb7d4-4gbj4_ffb99aaa-ca56-4f11-a26a-20d80226df6a/authorino/0.log"
Nov 02 02:36:05.970960 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:05.970925    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bjrd8"]
Nov 02 02:36:16.391070 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:16.390986    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4wp6c_b072a543-fc27-4360-bbf8-b3d41fcb468d/manager/0.log"
Nov 02 02:36:17.050507 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:17.050460    5639 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0320394b-gpzw4"]
Nov 02 02:36:17.100132 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:17.100090    5639 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0320394b-gpzw4"]
Nov 02 02:36:17.217169 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:17.217137    5639 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d03c8de-63cc-4af1-b612-05075329a1a8" path="/var/lib/kubelet/pods/1d03c8de-63cc-4af1-b612-05075329a1a8/volumes"
Nov 02 02:36:17.492658 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:17.492619    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65b55dd8bb-mccfz_b757d5cd-a65b-4f74-9ab5-4896809d8958/manager/0.log"
Nov 02 02:36:19.786105 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:19.786069    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-gqgr9_64171361-a2ca-468a-b389-26fb4a438fff/revoke-keys/0.log"
Nov 02 02:36:20.011770 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:20.011743    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-x4wrb_645900e9-c3f9-43f6-bd01-cb83dc083995/revoke-keys/0.log"
Nov 02 02:36:20.229433 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:20.229404    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-w6tql_fb81ec93-3689-4fc9-8ce0-8c17acbe88cc/revoke-keys/0.log"
Nov 02 02:36:21.993197 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:21.993154    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-595dbbb7d4-4gbj4_ffb99aaa-ca56-4f11-a26a-20d80226df6a/authorino/0.log"
Nov 02 02:36:23.855930 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:23.855897    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-bjrd8_ab8b43c4-5ac8-48bd-a379-35672a7b44ca/limitador/0.log"
Nov 02 02:36:25.699335 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:25.699300    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79f447648c-xvbbq_61991930-092d-41ce-855b-6ed5041e460a/kube-auth-proxy/0.log"
Nov 02 02:36:26.435051 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:26.435018    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-fd65x_faf2916d-803b-4f14-a2b1-1c45b742fc4d/istio-proxy/0.log"
Nov 02 02:36:27.420295 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:36:27.420242    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:36:27.713218 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:27.713140    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-q6wpx_db2e4537-de21-48f8-858e-1ead87636161/istio-proxy/0.log"
Nov 02 02:36:27.942928 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:27.942895    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-qnpm7_70ec0d83-9e6c-4e20-82b6-2f09909fc85f/istio-proxy/0.log"
Nov 02 02:36:28.167092 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:28.167060    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-whtg2_9e6c6d59-73b1-4b69-94d5-586b38902af4/istio-proxy/0.log"
Nov 02 02:36:28.667586 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:28.667550    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-smrvz_c574ce35-1740-4193-bcc3-38a38425b040/payload-pre-processing/0.log"
Nov 02 02:36:28.894467 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:28.894433    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-c276v_a75b836c-cfb5-4f2b-9a49-ae3b350b08d0/payload-processing/0.log"
Nov 02 02:36:30.051530 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:30.051497    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8_49af4db2-e792-4283-a393-47b5066464f8/main/0.log"
Nov 02 02:36:30.328735 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:30.328708    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-m8jkg_951363d0-8146-4109-8186-d232d1544131/main/0.log"
Nov 02 02:36:30.543474 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:30.543442    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-f8b2k_8eccb539-67a3-4801-98bb-93a1ae520121/main/0.log"
Nov 02 02:36:31.047411 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:31.047378    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh_6b06646f-8efd-4ebc-9b9c-ff8f9734da72/main/0.log"
Nov 02 02:36:31.562736 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:31.562702    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq_351876aa-c801-49f4-acdc-667422401ce3/main/0.log"
Nov 02 02:36:32.769455 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:32.769417    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-595dbbb7d4-4gbj4_ffb99aaa-ca56-4f11-a26a-20d80226df6a/authorino/0.log"
Nov 02 02:36:40.535236 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:40.535203    5639 scope.go:122] "RemoveContainer" containerID="bc3b5cfbcad6d58f519af33764b1562b36d341ccb0f317071d766671a24d5d22"
Nov 02 02:36:42.512843 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:42.512804    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4wp6c_b072a543-fc27-4360-bbf8-b3d41fcb468d/manager/0.log"
Nov 02 02:36:43.450954 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:43.450920    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65b55dd8bb-mccfz_b757d5cd-a65b-4f74-9ab5-4896809d8958/manager/0.log"
Nov 02 02:36:45.587620 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:45.587587    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-gqgr9_64171361-a2ca-468a-b389-26fb4a438fff/revoke-keys/0.log"
Nov 02 02:36:45.816083 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:45.816047    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-x4wrb_645900e9-c3f9-43f6-bd01-cb83dc083995/revoke-keys/0.log"
Nov 02 02:36:46.048907 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:46.048818    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-w6tql_fb81ec93-3689-4fc9-8ce0-8c17acbe88cc/revoke-keys/0.log"
Nov 02 02:36:47.478733 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:47.478691    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-595dbbb7d4-4gbj4_ffb99aaa-ca56-4f11-a26a-20d80226df6a/authorino/0.log"
Nov 02 02:36:49.159214 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:49.159178    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-bjrd8_ab8b43c4-5ac8-48bd-a379-35672a7b44ca/limitador/0.log"
Nov 02 02:36:50.775175 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:50.775140    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79f447648c-xvbbq_61991930-092d-41ce-855b-6ed5041e460a/kube-auth-proxy/0.log"
Nov 02 02:36:51.507836 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:51.507807    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-fd65x_faf2916d-803b-4f14-a2b1-1c45b742fc4d/istio-proxy/0.log"
Nov 02 02:36:52.731241 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:52.731211    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-q6wpx_db2e4537-de21-48f8-858e-1ead87636161/istio-proxy/0.log"
Nov 02 02:36:52.977034 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:52.977003    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-qnpm7_70ec0d83-9e6c-4e20-82b6-2f09909fc85f/istio-proxy/0.log"
Nov 02 02:36:53.210789 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:53.210749    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-whtg2_9e6c6d59-73b1-4b69-94d5-586b38902af4/istio-proxy/0.log"
Nov 02 02:36:53.702457 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:53.702427    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-smrvz_c574ce35-1740-4193-bcc3-38a38425b040/payload-pre-processing/0.log"
Nov 02 02:36:53.920652 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:53.920612    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-c276v_a75b836c-cfb5-4f2b-9a49-ae3b350b08d0/payload-processing/0.log"
Nov 02 02:36:54.800265 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:54.800233    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-hdcq8_49af4db2-e792-4283-a393-47b5066464f8/main/0.log"
Nov 02 02:36:55.074102 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:55.073998    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-m8jkg_951363d0-8146-4109-8186-d232d1544131/main/0.log"
Nov 02 02:36:55.357632 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:55.357597    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-f8b2k_8eccb539-67a3-4801-98bb-93a1ae520121/main/0.log"
Nov 02 02:36:55.829221 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:55.829187    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7czh_6b06646f-8efd-4ebc-9b9c-ff8f9734da72/main/0.log"
Nov 02 02:36:56.326506 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:36:56.326426    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-vvvjq_351876aa-c801-49f4-acdc-667422401ce3/main/0.log"
Nov 02 02:36:57.420173 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:36:57.420137    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:37:24.191008 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:37:24.190964    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65b55dd8bb-mccfz_b757d5cd-a65b-4f74-9ab5-4896809d8958/manager/0.log"
Nov 02 02:37:24.746115 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:37:24.746076    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65b55dd8bb-mccfz_b757d5cd-a65b-4f74-9ab5-4896809d8958/manager/0.log"
Nov 02 02:37:25.494937 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:37:25.494909    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-c276v_a75b836c-cfb5-4f2b-9a49-ae3b350b08d0/payload-processing/0.log"
Nov 02 02:37:25.720124 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:37:25.720093    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-c276v_a75b836c-cfb5-4f2b-9a49-ae3b350b08d0/payload-processing/0.log"
Nov 02 02:37:27.420654 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:37:27.420615    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:37:40.556760 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:37:40.556724    5639 scope.go:122] "RemoveContainer" containerID="eedb17cf096e33087ef6e3548cbe07bb9a2de26990b1c87ea6bb75947ca681bd"
Nov 02 02:37:40.566072 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:37:40.566046    5639 scope.go:122] "RemoveContainer" containerID="ab75c70cbed149161aa7241e1cfb31af333c2ca757a2a3342e13bbec3e491f40"
Nov 02 02:37:57.420563 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:37:57.420476    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:38:02.287396 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:02.287359    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65b55dd8bb-mccfz_b757d5cd-a65b-4f74-9ab5-4896809d8958/manager/0.log"
Nov 02 02:38:02.846550 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:02.846521    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65b55dd8bb-mccfz_b757d5cd-a65b-4f74-9ab5-4896809d8958/manager/0.log"
Nov 02 02:38:03.608206 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:03.608175    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-c276v_a75b836c-cfb5-4f2b-9a49-ae3b350b08d0/payload-processing/0.log"
Nov 02 02:38:03.838783 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:03.838754    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-c276v_a75b836c-cfb5-4f2b-9a49-ae3b350b08d0/payload-processing/0.log"
Nov 02 02:38:04.192977 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:04.192942    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zt4ds/must-gather-z2txl"]
Nov 02 02:38:04.193458 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:04.193443    5639 cpu_manager.go:414] "containerMap: removing container" podUID="645900e9-c3f9-43f6-bd01-cb83dc083995" containerName="revoke-keys"
Nov 02 02:38:04.193852 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:04.193459    5639 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="645900e9-c3f9-43f6-bd01-cb83dc083995" containerName="revoke-keys"
Nov 02 02:38:04.193852 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:04.193522    5639 memory_manager.go:365] "RemoveStaleState removing state" podUID="645900e9-c3f9-43f6-bd01-cb83dc083995" containerName="revoke-keys"
Nov 02 02:38:04.196706 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:04.196690    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zt4ds/must-gather-z2txl"
Nov 02 02:38:04.249726 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:04.249692    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zt4ds\"/\"kube-root-ca.crt\""
Nov 02 02:38:04.299185 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:04.299151    5639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zt4ds\"/\"openshift-service-ca.crt\""
Nov 02 02:38:04.299430 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:04.299369    5639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zt4ds\"/\"default-dockercfg-kqrg4\""
Nov 02 02:38:04.301029 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:04.301008    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zt4ds/must-gather-z2txl"]
Nov 02 02:38:04.359106 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:04.359068    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/16a28390-1b36-43af-8258-1b7f0b8aa7e2-must-gather-output\") pod \"must-gather-z2txl\" (UID: \"16a28390-1b36-43af-8258-1b7f0b8aa7e2\") " pod="openshift-must-gather-zt4ds/must-gather-z2txl"
Nov 02 02:38:04.359341 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:04.359147    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxjs2\" (UniqueName: \"kubernetes.io/projected/16a28390-1b36-43af-8258-1b7f0b8aa7e2-kube-api-access-nxjs2\") pod \"must-gather-z2txl\" (UID: \"16a28390-1b36-43af-8258-1b7f0b8aa7e2\") " pod="openshift-must-gather-zt4ds/must-gather-z2txl"
Nov 02 02:38:04.460320 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:04.460145    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxjs2\" (UniqueName: \"kubernetes.io/projected/16a28390-1b36-43af-8258-1b7f0b8aa7e2-kube-api-access-nxjs2\") pod \"must-gather-z2txl\" (UID: \"16a28390-1b36-43af-8258-1b7f0b8aa7e2\") " pod="openshift-must-gather-zt4ds/must-gather-z2txl"
Nov 02 02:38:04.460320 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:04.460255    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/16a28390-1b36-43af-8258-1b7f0b8aa7e2-must-gather-output\") pod \"must-gather-z2txl\" (UID: \"16a28390-1b36-43af-8258-1b7f0b8aa7e2\") " pod="openshift-must-gather-zt4ds/must-gather-z2txl"
Nov 02 02:38:04.467360 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:04.460608    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/16a28390-1b36-43af-8258-1b7f0b8aa7e2-must-gather-output\") pod \"must-gather-z2txl\" (UID: \"16a28390-1b36-43af-8258-1b7f0b8aa7e2\") " pod="openshift-must-gather-zt4ds/must-gather-z2txl"
Nov 02 02:38:04.521675 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:04.521640    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxjs2\" (UniqueName: \"kubernetes.io/projected/16a28390-1b36-43af-8258-1b7f0b8aa7e2-kube-api-access-nxjs2\") pod \"must-gather-z2txl\" (UID: \"16a28390-1b36-43af-8258-1b7f0b8aa7e2\") " pod="openshift-must-gather-zt4ds/must-gather-z2txl"
Nov 02 02:38:04.811399 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:04.811312    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zt4ds/must-gather-z2txl"
Nov 02 02:38:05.184930 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:05.184843    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zt4ds/must-gather-z2txl"]
Nov 02 02:38:05.187435 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:38:05.187405    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16a28390_1b36_43af_8258_1b7f0b8aa7e2.slice/crio-bdb9dc0dbe1619efa7ffa1fd1491aafdc64f9a028c113503c3e2e73e04b033aa WatchSource:0}: Error finding container bdb9dc0dbe1619efa7ffa1fd1491aafdc64f9a028c113503c3e2e73e04b033aa: Status 404 returned error can't find the container with id bdb9dc0dbe1619efa7ffa1fd1491aafdc64f9a028c113503c3e2e73e04b033aa
Nov 02 02:38:05.189353 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:05.189335    5639 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 02:38:05.874695 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:05.874656    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zt4ds/must-gather-z2txl" event={"ID":"16a28390-1b36-43af-8258-1b7f0b8aa7e2","Type":"ContainerStarted","Data":"bdb9dc0dbe1619efa7ffa1fd1491aafdc64f9a028c113503c3e2e73e04b033aa"}
Nov 02 02:38:06.882397 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:06.882339    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zt4ds/must-gather-z2txl" event={"ID":"16a28390-1b36-43af-8258-1b7f0b8aa7e2","Type":"ContainerStarted","Data":"44f2871b1388f6ef8a461ff55155be1d7fef521b1318498598fa97e0a8437634"}
Nov 02 02:38:06.882819 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:06.882406    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zt4ds/must-gather-z2txl" event={"ID":"16a28390-1b36-43af-8258-1b7f0b8aa7e2","Type":"ContainerStarted","Data":"79cd16817be0113d94e7f1bc66de4d87e2a950715c70f75e09bc0d64c9d8da2d"}
Nov 02 02:38:06.998055 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:06.998003    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zt4ds/must-gather-z2txl" podStartSLOduration=1.714792111 podStartE2EDuration="2.997986795s" totalImagesPullingTime="1.283194684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:38:06.99724408 +0000 UTC m=+4351.900926196" watchObservedRunningTime="2026-11-02 02:38:06.997986795 +0000 UTC m=+4351.901668909"
Nov 02 02:38:14.261299 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:14.258166    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9kqsh_26cf53e3-9cf3-4ad5-afe2-5c241f6749ad/global-pull-secret-syncer/0.log"
Nov 02 02:38:17.314330 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:17.312540    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vlg9k_929a4d2e-02fa-4d5f-ace0-0ad3d41614b7/konnectivity-agent/0.log"
Nov 02 02:38:18.967590 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:18.966597    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-53.us-west-1.compute.internal_14b7d72c2c054e228c464e14811dfb58/haproxy/0.log"
Nov 02 02:38:23.748976 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:23.748784    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-595dbbb7d4-4gbj4_ffb99aaa-ca56-4f11-a26a-20d80226df6a/authorino/0.log"
Nov 02 02:38:24.450424 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:24.450393    5639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zt4ds/perf-node-gather-daemonset-lc9qp"]
Nov 02 02:38:24.457532 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:24.457501    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-lc9qp"
Nov 02 02:38:24.546475 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:24.546438    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zt4ds/perf-node-gather-daemonset-lc9qp"]
Nov 02 02:38:24.618073 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:24.618039    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d-lib-modules\") pod \"perf-node-gather-daemonset-lc9qp\" (UID: \"ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-lc9qp"
Nov 02 02:38:24.618268 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:24.618098    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d-sys\") pod \"perf-node-gather-daemonset-lc9qp\" (UID: \"ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-lc9qp"
Nov 02 02:38:24.618268 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:24.618217    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwhds\" (UniqueName: \"kubernetes.io/projected/ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d-kube-api-access-zwhds\") pod \"perf-node-gather-daemonset-lc9qp\" (UID: \"ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-lc9qp"
Nov 02 02:38:24.618268 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:24.618255    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d-podres\") pod \"perf-node-gather-daemonset-lc9qp\" (UID: \"ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-lc9qp"
Nov 02 02:38:24.618426 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:24.618374    5639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d-proc\") pod \"perf-node-gather-daemonset-lc9qp\" (UID: \"ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-lc9qp"
Nov 02 02:38:24.719999 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:24.719912    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d-lib-modules\") pod \"perf-node-gather-daemonset-lc9qp\" (UID: \"ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-lc9qp"
Nov 02 02:38:24.719999 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:24.719972    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d-sys\") pod \"perf-node-gather-daemonset-lc9qp\" (UID: \"ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-lc9qp"
Nov 02 02:38:24.720222 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:24.720018    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwhds\" (UniqueName: \"kubernetes.io/projected/ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d-kube-api-access-zwhds\") pod \"perf-node-gather-daemonset-lc9qp\" (UID: \"ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-lc9qp"
Nov 02 02:38:24.720222 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:24.720048    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d-podres\") pod \"perf-node-gather-daemonset-lc9qp\" (UID: \"ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-lc9qp"
Nov 02 02:38:24.720222 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:24.720056    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d-lib-modules\") pod \"perf-node-gather-daemonset-lc9qp\" (UID: \"ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-lc9qp"
Nov 02 02:38:24.720222 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:24.720112    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d-sys\") pod \"perf-node-gather-daemonset-lc9qp\" (UID: \"ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-lc9qp"
Nov 02 02:38:24.720222 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:24.720123    5639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d-proc\") pod \"perf-node-gather-daemonset-lc9qp\" (UID: \"ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-lc9qp"
Nov 02 02:38:24.720222 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:24.720125    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d-podres\") pod \"perf-node-gather-daemonset-lc9qp\" (UID: \"ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-lc9qp"
Nov 02 02:38:24.720222 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:24.720180    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d-proc\") pod \"perf-node-gather-daemonset-lc9qp\" (UID: \"ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-lc9qp"
Nov 02 02:38:24.781507 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:24.781469    5639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwhds\" (UniqueName: \"kubernetes.io/projected/ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d-kube-api-access-zwhds\") pod \"perf-node-gather-daemonset-lc9qp\" (UID: \"ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-lc9qp"
Nov 02 02:38:25.074806 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:25.074719    5639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-lc9qp"
Nov 02 02:38:25.463961 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:25.463934    5639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zt4ds/perf-node-gather-daemonset-lc9qp"]
Nov 02 02:38:25.467336 ip-10-0-136-53 kubenswrapper[5639]: W1102 02:38:25.467296    5639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podff3abaeb_f7ac_4ff3_b9b4_61e10cc62f7d.slice/crio-d4b6d2f3e2318d578e2a0565f0e60467b75f5989b668d126de05cf543353a698 WatchSource:0}: Error finding container d4b6d2f3e2318d578e2a0565f0e60467b75f5989b668d126de05cf543353a698: Status 404 returned error can't find the container with id d4b6d2f3e2318d578e2a0565f0e60467b75f5989b668d126de05cf543353a698
Nov 02 02:38:25.751839 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:25.751750    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-bjrd8_ab8b43c4-5ac8-48bd-a379-35672a7b44ca/limitador/0.log"
Nov 02 02:38:26.032258 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:26.032176    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-lc9qp" event={"ID":"ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d","Type":"ContainerStarted","Data":"64e799f410e0c7408ec37b4a85bedc9bba40b1bd91149602b41833c01ec3c96c"}
Nov 02 02:38:26.032258 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:26.032240    5639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-lc9qp" event={"ID":"ff3abaeb-f7ac-4ff3-b9b4-61e10cc62f7d","Type":"ContainerStarted","Data":"d4b6d2f3e2318d578e2a0565f0e60467b75f5989b668d126de05cf543353a698"}
Nov 02 02:38:26.032727 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:26.032312    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-lc9qp"
Nov 02 02:38:26.148524 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:26.147961    5639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-lc9qp" podStartSLOduration=2.1479408700000002 podStartE2EDuration="2.14794087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:38:26.147188042 +0000 UTC m=+4371.050870147" watchObservedRunningTime="2026-11-02 02:38:26.14794087 +0000 UTC m=+4371.051622986"
Nov 02 02:38:27.425926 ip-10-0-136-53 kubenswrapper[5639]: E1102 02:38:27.425887    5639 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrpsb" containerName="extractor"
Nov 02 02:38:28.779252 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:28.779225    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8aa5a540-cf10-47bc-954c-a92d8418f7e7/alertmanager/0.log"
Nov 02 02:38:29.194824 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:29.194796    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8aa5a540-cf10-47bc-954c-a92d8418f7e7/config-reloader/0.log"
Nov 02 02:38:29.610030 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:29.609981    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8aa5a540-cf10-47bc-954c-a92d8418f7e7/kube-rbac-proxy-web/0.log"
Nov 02 02:38:30.024551 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:30.024471    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8aa5a540-cf10-47bc-954c-a92d8418f7e7/kube-rbac-proxy/0.log"
Nov 02 02:38:30.439696 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:30.439669    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8aa5a540-cf10-47bc-954c-a92d8418f7e7/kube-rbac-proxy-metric/0.log"
Nov 02 02:38:30.853078 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:30.853047    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8aa5a540-cf10-47bc-954c-a92d8418f7e7/prom-label-proxy/0.log"
Nov 02 02:38:31.268836 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:31.268750    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8aa5a540-cf10-47bc-954c-a92d8418f7e7/init-config-reloader/0.log"
Nov 02 02:38:32.050112 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:32.050085    5639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-lc9qp"
Nov 02 02:38:32.205044 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:32.205012    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-g7f8f_de6949bb-b875-4ca6-b0d3-19c07290bfae/kube-state-metrics/0.log"
Nov 02 02:38:32.615733 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:32.615677    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-g7f8f_de6949bb-b875-4ca6-b0d3-19c07290bfae/kube-rbac-proxy-main/0.log"
Nov 02 02:38:33.031538 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:33.031433    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-g7f8f_de6949bb-b875-4ca6-b0d3-19c07290bfae/kube-rbac-proxy-self/0.log"
Nov 02 02:38:33.450569 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:33.450542    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5bfddd89bf-5vgk2_b49b523f-d706-4d6c-aac0-020b2e75c88a/metrics-server/0.log"
Nov 02 02:38:34.289188 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:34.289157    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s59gt_ac453c3c-9209-4c20-8fe7-07c6f561b855/node-exporter/0.log"
Nov 02 02:38:34.702949 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:34.702913    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s59gt_ac453c3c-9209-4c20-8fe7-07c6f561b855/kube-rbac-proxy/0.log"
Nov 02 02:38:35.123465 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:35.123437    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s59gt_ac453c3c-9209-4c20-8fe7-07c6f561b855/init-textfile/0.log"
Nov 02 02:38:38.071803 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:38.071772    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-cj277_bc6acc82-57d4-4906-9c57-67d717e89ba3/kube-rbac-proxy-main/0.log"
Nov 02 02:38:38.487384 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:38.487352    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-cj277_bc6acc82-57d4-4906-9c57-67d717e89ba3/kube-rbac-proxy-self/0.log"
Nov 02 02:38:38.906269 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:38.906240    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-cj277_bc6acc82-57d4-4906-9c57-67d717e89ba3/openshift-state-metrics/0.log"
Nov 02 02:38:40.590025 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:40.589988    5639 scope.go:122] "RemoveContainer" containerID="949bf67d7dbdf98ee5e1125d150772990393483384f42a33f569874ddecde3f6"
Nov 02 02:38:43.504791 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:43.504748    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bdd7cc4cc-566sl_27bd836b-2ab9-462c-8e6d-63fe3d090d04/telemeter-client/0.log"
Nov 02 02:38:43.918955 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:43.918918    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bdd7cc4cc-566sl_27bd836b-2ab9-462c-8e6d-63fe3d090d04/reload/0.log"
Nov 02 02:38:44.336359 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:44.336328    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bdd7cc4cc-566sl_27bd836b-2ab9-462c-8e6d-63fe3d090d04/kube-rbac-proxy/0.log"
Nov 02 02:38:44.758244 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:44.758164    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c68bc9867-x88cx_f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/thanos-query/0.log"
Nov 02 02:38:45.171480 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:45.171442    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c68bc9867-x88cx_f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/kube-rbac-proxy-web/0.log"
Nov 02 02:38:45.586007 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:45.585980    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c68bc9867-x88cx_f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/kube-rbac-proxy/0.log"
Nov 02 02:38:46.001826 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:46.001743    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c68bc9867-x88cx_f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/prom-label-proxy/0.log"
Nov 02 02:38:46.416713 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:46.416681    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c68bc9867-x88cx_f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/kube-rbac-proxy-rules/0.log"
Nov 02 02:38:46.831463 ip-10-0-136-53 kubenswrapper[5639]: I1102 02:38:46.831419    5639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c68bc9867-x88cx_f11d2490-7d7e-4ce5-bdd1-8f30ebe09e1d/kube-rbac-proxy-metrics/0.log"